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(57) Abstract 

A Cooperative Help Assistance (CHA) system and method provide real-time user assistance for one or more windows-based Graphic 
User Interface (GUI) applications or a single application's different subsections such as web pages, running concurrently in any operating 
system. The CHA System enables the development of an informative assistance object (415) independently from the original source 
code or development environment of the target Host Application. The assistance object can be selected by any number of user interfaces 
from sophisticated inference driven interactive interface search tools or categorized lists. By intercepting and monitoring user actions on 
a Host Application, the CHA system performs intelligent assistance in the context of the target host application program. Utilizing a 
Host Application Model, the CHA system and method dynamically assemble many elements (405, 410, 420, 435, 440) in real-time or 
just-in-time to produce assistance sequences or elements very efficiently without having to code every interface path permutation. 
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SYSTEM AND METHOD FOR DYNAMIC ASSISTANCE IN SOFTWARE 
APPLICATIONS USING BEHAVIOR AND HOST APPLICATION MODELS 

COPYRIGHT NOTICE 
5 A portion of the disclosure of this patent document contains material which is 

subject to copyright protection. The copyright owner has no objection to the facsimile 
reproduction by anyone of the patent document or the patent disclosure, as it appears in the 
U.S. Patent and Trademark Office patent files or records, but otherwise reserves all copyright 
rights whatsoever. 
10 BACKGROUND OF THE INVENTION 

The present invention relates to software for assisting users, and, more 
particularly, to a system and method for interactively assisting a user operating an application 
program. 

1 5 The rapid rate of development and proliferation of software throughout work, 

business transactions, and society have required a high degree of learning and absorption by 
users to properly operate and gain the advantage of such software. However, as more and 
more features are integrated into application software, the complexity and shear volume of 
diverse features can be overwhelming, even for experienced computer users. In addition, 

20 interfaces for applications have increased the possibility of confusion to the users in 

attempting to provide application operators with more choices for improved productivity. For 
example, stacked dialog boxes and a multitude of taskbars, icons, commands, and options 
have given users the opportunity to be more productive, but the very same increase in 
operational options has decreased the ability of a user to learn and thoroughly understand 

25 such application software. 

Since software and application programs are being dynamically improved and 
distributed, the learning environment for a user is constantly and rapidly changing. 
Accordingly, a need exists for development tools and assistance software which guide a user 
through such complex and changing software environments. 

30 In addition, computer applications typically provide exceedingly repetitive and 

uniform interfaces which, for example, confuse users by having icons appearing similar when 

having different functions in different applications. A need exists for an interface and/or an 

1 
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assistance system and method for guiding a user through different applications to properly 
use any given application despite similar application interfaces and commands. 

Developments in accessing data for timely delivery have increased with more 
networking between computing environments, such as the Internet. World Wide Web 
5 (WWW) browsers and search engines as well as self-service and automation such as Internet- 
traversing bots and push technology have increased the productivity of users. However, such 
network-based environments also suffer from the indicated interface complexity issues. 

Separate from application developers, third-party or remote software 
developers have developed assistance systems which aid users to use application programs 

10 with greater ease and productivity, for example, to quickly access the typically 5 % of an 
application program most commonly used by typical users. Such assistance systems may be 
capable of working with the application program without accessing the internal operation of 
the application program code itself. Thus, separate assistance systems may enhance 
productivity and reduce the complexity of the underlying host application program. 

15 However, in the prior art, such assistance systems are limited to generating 

help panes and dialog boxes, which occupy space on a display, which is relatively "valuable" 
from an interface-based perspective. For example, U.S. Patent No. 5,825,356 describes a 
help system with semi-transparent windows for disabling controls, with the help system 
disabling buttons and interface screen icons and generating a static balloon to highlight 

20 portions of the interface. However, such highlighting balloons tend to clutter the displayed 
interface, and so are not effective in assisting users. 

In addition, help or assistance systems and methods for software applications 
generally do not use overlays for display on the same interface screen as the software 
application, and do not have multimedia operations, so such prior art systems are not effective 

25 in assisting users. Furthermore, known help systems are pre-coded using, for example, script 
languages to handle predetermined commands and user actions, and so are statically fixed and 
incapable of run-time path determination to interactively assist a user to learn an application 
program. Such script languages also prevent third-party developers from rapidly creating 
new or improved production tools as the underlying application program is changed. 

30 Moreover, help systems are primarily created for single user operation, and so are not 
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typically networked or network-compatibly for collaborative or real-time shared use by many 

users learning or using a common application program. 

SUMMARY 

A Cooperative Help Assistance (CHA) system and method provide real-time 
5 user assistance for one or more windows-based Graphic User Interface (GUI) applications or 
a single application's different subsections such as web pages, running concurrently in any 
operating system. The CHA System enables the development of an informative assistance 
object (obj) independently from the original source code or development environment of the 
target Host Application. The assistance object can be selected by any number of user 
10 interfaces from sophisticated inference-driven interactive interface search tools or categorized 
lists. By intercepting and monitoring a user's actions on a Host Application, the CHA system 
and method perform intelligent assistance in the context of the target host application 
program. Utilizing a specialized CHA Database which contains a Host Application Model, 
and numerous behavioral, resource and information objects, the CHA System and method 
15 dynamically assembles many elements in real-time or just-in-time operation, which makes the 
production of assistance sequences or elements efficient without having to necessarily code 
every interface path permutation. Paths can be dynamically generated from the Host App. 
Model. 

Furthermore, the Model enables a real-time module to offer intelligent, 
20 contextual assistance as well as the real-time construction of automated, accelerated CHA 
Sequences or Procedures that require little or no user interaction. All pertinent assistance and 
information are processed and expressed by an extensive multitasking, multimedia subsystem 
in both two dimensional (2D) and real-time three-dimensional (3D) application interfaces, 
which greatly enhances and extends the effectiveness of any explanation or material 
25 expression. The production of Assistant Sequences is facilitated by the Host App. Model and 
2D and 3D GUI "drag and drop" interface tools and in this mode does not require scripting 
languages or predetermined static programming. Scripting, however, is optionally accessible 
for specific or specialized tasks. By being adaptive across any number of processes and 
applications and different environments such as networks, the Internet, or stand-alone 
30 applications, the CHA System and method provide alternative and rapid solutions for the 

3 
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problem of presenting indirect, lengthy text explanations about using GUI applications and 
their often complex, abstract interfaces and procedures. 
BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 shows an example of a computer system including hardware and 
5 software for implementing the present invention; 

FIG. 1 A shows portions of the computer system of FIG. 1 in greater detail 
illustrating different run-time components and some of the development tools used in analysis 
and the constructing of a CHA system and method; 

FIG. 2 shows an example of a typical application and interface for opening a 
10 file via a dialog box; 

FIG. 3 shows the main Modes of the disclosed CHA System and method of 
the present invention; 

FIG. 4 shows a CHA Interface Overlay with a Guide Character and Text 

InfoObjects; 

15 FIG. 5 shows a CHA Interface Overlay with an Adjunct Window where 

complex animated interactive activities run independently and communicate bi-directionally 
with a Host Application. 

FIG. 6 illustrates CHA Activity With Host App GUI Linkage; 

FIG. 6A shows an ICAP control map; 
20 FIG. 7 is an overview of the CHA run-time engine modules; 

FIG. 8 illustrates CHA Run-Time Component Initialization; 

FIG. 9 illustrates a CHA Database (ChaDB); 

FIG. 10 illustrate Host Interface GUI Objects and Data Structures; 

FIG. 1 1 illustrates InfoObj Data Structures; 
25 FIG. 12 illustrates Guide Sequence Processing Modules; 

FIG. 13 illustrates Dynamic and Fixed Guide Sequence Processing; 

FIG. 14 illustrates Fixed Guide Sequence ChaSys Object Set Execution; 

FIG. 14A illustrates Dynamic Guide Sequence Processing in greater detail; 

FIG. 15 illustrates CHA Monitor Collection Processing; 
30 FIG. 15A illustrates CHA Monitor and Dynamic Sequence Preparation; and 

FIG. 16 illustrates Client/Server Interaction for Dynamic Guide Sequences. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

FIG. 1 illustrates a client computer and server for implementing and operating 
the disclosed Cooperative Help Assistance (CHA) system and method on or with a client 
computer. For example, the CHA system and method may be an application program or 

5 software executed by the client computer, which may be a personal computer, workstation, or 
other computing platforms. The Server 105, another computer distinguished by its server 
software, distributes data to many client computers and complements client processing tasks. 
The Network Connector 1 1 0 provides connectivity to external computers primarily servers. 
A standard connection protocol transmits and receives data in any format between remote 

10 computers. The client computer may include a number of subsystems connected together by 
an internal system bus. Instructions included in the system RAM 130 are executed by the 
Central Processing Unit 1 50. 

Numerous external devices are connected to the Input/Output (I/O) Controller 
1 15 via Ports 140 which provide a connection to external peripherals such as a modem on a 

15 serial port. The display of 2D and 3D graphics and video or any form of moving images is 
handled by the Video Adapter 120 that displays the graphics buffer on the Monitor 125. 
Sound processing is performed by the Sound Hardware 135 which may be capable of 
handling all standard operating system formats, such as a music industry standard such as 
Music Interface Digital Interface (MIDI), as well as sound or speech, text-to-speech 

20 capability, CD Audio, or any other known sound format. The Pointing Device 1 55 and 
Keyboard 145 transform user inputs into input data, with such user inputs generated from a 
user interacting with visual information as displayed. Permanent Storage 160 retains data 
between computer power off states, and is accessible, as needed, at run-time. 

As shown in FIG. 1 A, the disclosed CHA system 1 309 and method may be 

25 used in a computing environment in conjunction with a Host Application 1318 using, for 
example, a graphic user interface (GUI) or other known application interfaces, such as 
"MICROSOFT WINDOWS" 1315. The Host Application receives an injected CHA Hook 
component 1321 that intercepts and analyses operating system messages before generating 
events that are passed to the CHA Engine 1324. Furthermore, the CHA System 1309 and 

30 method include user and developer run-time components such as a CHA engine 1324, CHA 
Operational Code 1336, and an Interactive Virtual Machine (IVM) engine 1330 and IVM 
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Code 1 333. The IVM component is the Expression Engine which renders all visual and aural 
processes including animation, text, digitized speech, or any visual or aural feedback during 
the execution of assistance solutions. Data and other inputs may be received through the 
input devices 1315, such as a mouse, keyboard, a CD-ROM 1315, and network connections 

5 and protocols 1320, such as connections and interfaces to the Internet, the World Wide Web, 
and/or other networks and on-line services. 

The CHA system 1309 may also operate in conjunction with CHA 
Development Environment (CDE) 1312. This CDE contains a number of tools: a GUI 
Analyzer 1342, a Host App Model Builder 1343, IVM multimedia Authoring, and the CHA 

1 0 Studio which integrates data from 1 342, 1343, and 1 345 to produce the output CHA file 
1351 . The CHA system 1309 and the operating system 1306 with the Host Application 1318 
may be implemented, for example, on an "INTEL PENTIUM"-based system using 
"MICROSOFT WINDOWS NT' to provide the GUI with known input/output devices, 
including a display, a keyboard, a mouse, etc. 

15 The appearance of portions of a display may include, for example, the color, 

the shape, and/or the location of such portions. 

Regions of the display corresponding to, for example, commands may have 
distinct and/or substantially defined boundaries from other portions of the display. Such 
regions may include icons, radio buttons, windows, task bars, tool bars, and the like. 

20 The current screen position indicator (CSPI) is displayed on the GUI to relate 

a specifically active region of the display to the user. In particular, the CSPI may be a cursor 
of any shape and/or size, such as an arrow, or any other specific region of the display which is 
responsive to inputs from the user to reflect an active region for pointing and/or actuation by 
the user. For example, the CSPI may be responsive to user inputs corresponding to 

25 movements of a mouse device or a pointing device such as track pads or a light pen. 

Touchscreen inputs and infrared signals from remote devices may also be used to control the 
CSPI. 

A region on the display is actuatable if, in response to a first input from the 
user having the CSPI associated with the region and in response to a second input of the user, 
30 a command or data entry is initiated. The region may be, for example, a clickable icon and/or 
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radio button which the user may actuate by positioning, for example, a cursor "over" the icon 
or button and by pressing a mouse button. 

Actuation of an icon or other interface items may include clicking a button on 
a mouse, such as a left or right button. 

5 The CHA system and method uses a Host App Model that describes logically 

the entire interface, or subset thereof, of a Host Application, and facilitates many automated 
processes such as real-time assembly of the presentation of information or automatic 
generation of accelerated procedures. 

FIG. 2 illustrates a typical windows-based application having a main window, 

10 numerous child windows, and Graphic User Interface (GUI) objects or GuiObjs which are the 
application's interface objects that the user interacts with. This example shows a typical 
application written in the "MICROSOFT WINDOWS" environment and is also 
representative of any windowed GUI-based operating system such "X WINDOWS", the 
"APPLE MACINTOSH", and many others. In this illustration one can view the various 

15 GuiObjs which are the two-dimensional (2D) representations of various GuiObjs that react to 
user input from the keyboard, a pointing device such as the mouse, or any other input device. 
The title of the application is indicated at in the application's title bar 205, which is a GuiObj 
used to move the application's main window to different screen locations. More "global" 
control of the application such as minimizing the window, Changing window modes, etc. is 

20 done with the GuiObjs 220. Other GuiObjs of different application functions are located in 
various parts of the application window such as the Menu items 210 and the Toolbar items 
215. In this example an OpenFile dialog box 230 has been opened in the main window 235, 
with the OpenFile dialog box 230 having been invoked by a GuiObj in the menu interface 
section 210. 

25 Furthermore, the dialog box 230 determines different file functions, in this 

case a loading of an animation file type i.e. having a filename suffix *.a using the input field 
240. Other GuiObjs are shown, such as the cancel icon 225 and the file type field 240. The 
different GuiObjs of different types are used as elements to determine CHA Sequences for 
performing the CHA operations. A Sequence is a sequential, step-by-step presentation of 

30 information via InfoObjs synchronized with the user's interactions with a Host App's 
GuiObjs. An InfoObj contains all the vital information being presented to the user. Any 
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windows application, including any Internet browser, is a potential Host Application that can 
interact with the CHA System. 

Referring to FIG. 3, the various modes of operation of the CHA system and 
method are shown in relation to each other. The Guide Mode 305 performs step-by-step 
5 procedures are performed to be a basic guide to a new user. Additionally graphically 
expressive tutorials using multiple graphical, animated expressions can be used to lead the 
user through complex procedures. The Interactive Custom Accelerated Procedure (ICAP) 
Mode 315 provides interactive, customizable procedures that greatly accelerate the use of an 
application to increase productivity of the experienced user. In Activity Mode 320, an 

10 Adjunct Window is generated which can provide complex animated interactive activities that 
can be independent or that can communicate directly with the interface of a Host Application. 
Such operations in Activity Mode complement the user interface (UI) object overlays with 
extensive, additional detailed information. 

Furthermore, the CHA System is available in two basic forms or versions: a 

15 User Environment 330 and Development Environment 335. The User version 330 is the user 
run-time version of the application that is released to the general user, while the Development 
version 335 is the form of the assisting application in which a CHA delivery application is 
created and its data assembled by the Application Expert. Tools that synchronize and target 
specific actions in the Host Application facilitate such delivery application creation and 

20 assembly. 

FIG. 3 also shows the main divisions of operation of the CHA System having 
main groups: a first group having the principal modes of operation 305, 315, 320, 325, 327, 
while the second group includes the type of run-time versions i.e. Development vs. User. The 
high level CHA System Manager 310 acts as the master controller, which routes the 
25 processing for the setting of or exiting from different modes, and which keeps track and 
controls the overall state of the CHA application. Other functions might pass important data 
between the different modes or toggle between run time execution and asynchronous 
development actions. 

In the first group, there are the five principal Modes of Operation: Guide, 
30 ICAP, Activity, Monitor, and Explore. In the first, the Guide Mode 305, the CHA System 
offers step-by-step guidance to any procedure of a Host Application on the live application. 

8 
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This mode 305 also includes tutorial-like activities during which the user participates in 
interaction with underlying logic, and the mode 305 includes processes that assesses the 
user's knowledge of the application. For example, after viewing the steps of a procedure, the 
program prompts the user to recall all the steps. As the user inputs each step, the CHA engine 
5 monitors and assesses the user's actions along the prescribed goal, and offers hints via an 
optional Guide Character or Character Group Set e.g. two or more characters in an ensemble, 
which are displayed to the user and which may interact with each other as well, to provide 
encouragement or discouragement, in a manner similar to a live teacher with or without a . 
sense of humor. 

10 All of the above activities in the guide mode 305 present material to the user in 

a wide variety of expressions from straightforward text to multiple-object animated objects, 
to show time or spatial relationships as pertinent to the procedure being used. The use of 
animation and concurrent processes can greatly enhance the learning process far beyond static 
textual forms. 

15 The second mode is the ICAP mode 315, which provides control of the Host 

Application in an accelerated mode of operation such that the user performs dramatically 
fewer actions of an input device, such as the mouse or keyboard, to reach and use any 
functional parts of the application, no matter how deeply buried such functional parts are 
located in the interface structure. Facilities are provided to customize the procedure i.e. 

20 define the steps, goals and destinations, as well as to interact during the procedure. For 
example, if the user needs to make a selection during one of the steps of the procedure, the 
user is allowed to make the selection in the ICAP mode 3 15, as opposed to the common 
"macro" facilities found in many applications in which the user records steps or actions but 
cannot make choices during the operation. In conjunction with the Host App Model, the user 

25 can very rapidly navigate to any part of the program. Furthermore, ICAP can be considered a 
Guide Sequence process except with InfoObjs and Commentary objects suppressed where 
many Host Application controls, Host App Model interaction, operating system controls, and 
other related CHA System mechanisms being similar or identical to the Guide Sequence. 
One skilled in the art would recognized that an adaptation from Guide Mode Fixed or 

30 Dynamic types to ICAP Mode mechanisms may be readily implemented. 
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The third mode of operation is Activity Mode 320. Although this mode 320 
may appear to be removed from the Host Application, this is not necessarily the case. First, 
this mode uses an Adjunct Window 320 in which a wide variety of expressions or 
interactivity can be performed, ranging from static text or graphics, to fully interactive, lively 

5 and animated multimedia-rich applications including rapid paced games. In Activity Mode, 
the objects in the Adjunct Window are linked or associated with the live application. For 
example, if the user drags and drops an object inside the Adjunct Window, an event can be 
generated that sends a message to the Host Application, and one of its GuiObjs may then be 
activated such as to perform the opening of a dialog box. 

10 The term "GuiObj" used herein refers to a logical, virtual entity in the CHA 

System which is used for the Host App Model representation. The actual "physical" 
application interface controls are managed by the operating system's internal data structures 
e.g. window handles or window class structures, etc. In the CHA System, GuiObjs have 
associative SysGuiObjs that enable respective mechanisms for the manipulation or internal 

15 interaction with interface objects that are manifested to the user. When the user is interacting 
with a GuiObj, the SysGuiObj is inextricably bound together with the GuiObj. In reference 
to an interaction with a GuiObj, the SysGuiObj i.e. the actual "physical" properties provided 
by the operating system and hence, its interfacing with input hardware, is implicit. For 
example to say that a user clicks on the GuiObj, the user is actually interacting with the 

20 "physical properties of the operating system but concurrently the GuiObj is placed into a 
context within the Host Application Model via mechanisms that use it. 

Additionally, events occurring or being performed in the Host Application can 
also generate a message to an object inside the Adjunct Window application to activate an 
animation that illustrates an underlying process occurring in the Host Application. In general, 

25 the Adjunct Window complements the Host Application and augments the expression output 
to the user for the explaining of a Host Application's organizational concepts or principles 
and details. With the Adjunct Window, complex animated interactive activities may be 
provided which can be either independent or linked i.e. communicate directly with the 
interface of a Host Application. 

30 The fourth mode shown in FIG. 3 is the Monitoring Mode 325 which collects 

all user input into a database for either real-time or post-operation analysis. In this mode 325, 

10 
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detection of usage patterns determines the presentation of different options or solutions to the 
user for more efficient use of the Host Application program, which may be in context such 
that, as the user indicates what the overall goal is, the Monitoring Mode 325 can use an 
Inference Processor in conjunction with ChaDB's Host App Model to suggest what the next 

5 steps may be, or alternatively present other options in different areas of the program. During 
every use or session of the Host App program, the CHA system collects a user data set. After 
the user stops using the Host App program for a given session, post-operation analysis is 
performed to determine any patterns from the user data set to generate a usage report. 
Furthermore, from the Post Analysis, ICAP procedures can be built automatically for the user 

10 and placed in a list for later access. 

The fifth mode shown in FIG. 3 is the Explore Mode 327, which allows the 
user to randomly access any part of the entire interface of the Host Application, and to receive 
embedded InfoObj presentations stored in application's ChaDB. 

In FIG. 3, the Development 335 and User 330 versions of the CHA System 

15 relate to the development time vs. actual run-time form of the data and code that is available 
to the user. In the system, the distinction between User and Development is naturally related 
to the access rights of the user to certain functionality and to the provision of a set of probing 
tools for analyzing a Host Application and for preparing the assistance functionality for the 
targeted program. 

20 The CHA system provides a mechanism to coordinate and synchronize events 

across multiple application processes, for example, to begin a CHA sequence on the browser 
followed by events taking place on one or more applications, and then returning again to the 
browser. The CHA System Manager 3 1 0 handles this coordination of CHA Events and 
Messages. The user, upon requiring assistance or wanting guidance about a particular topic, 

25 can select from a list of choices provided by the CHA system. 

FIG. 4 shows an example application with two types of assistance 
enhancements i.e. text and animated objects, GuiObj Highlighting 405 and an interactive 
Guide Character 41 5. The Guide Character provides a point of focus for the user during use 
of the Host Application and the CHA system to provide a friendly personification or image to 

30 describe the Host application's logic or operations. The first Highlighting Effect 405 

associated with the Animation Menu is implemented using moving graphics, as suggested in 

11 
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FIG. 4 with the nested outer rectangles 410 which represent multiple, animated frames. The 
second Highlighting Effect 420 is another indicator that directs the user to an interface object 
as another part of the presentation sequence, and the second effect 420 is triggered by a 
stream of events which synchronize the Guide Character 415. The Guide Character 41 5 can 
5 also be displayed in conjunction with the processing of a sound or a voice file to aurally 
convey information using an audio-based mechanism of the operating system and/or 
computer system running the Host Application program and the CHA system. The CHA 
System can control any number of multitasking objects such as Highlights 405, 410, 420 of 
the GuiObjs because of its multitasking architecture, and the CHA System can maintain 

10 synchronization over all presented events, user input events, and other streamed events 
produced by the Guide Character's underlying logic. 

An InfoObj is attached to and/or associated with the Highlighted Interface 
GuiObjs 405, 410, 420 such that, if the user moves the interface window 430, the display of 
the InfoObj remains attached and follows the window 430. It is important to note that the 

15 term InfoObj refers to all the essential information being conveyed in whatever format. Such 
a format could be visual i.e. text whether static or animated, graphic representations, in 2 or 3 
dimensional form, or aural i.e. digitized speech. Or in combination i.e. video, interactive 
illustrations, and so on. 

The CHA system uses Overlays, which are User Interface Overlays in 

20 combination with InfoObj presentations or overlays as well as the Guide Character presented 
as overlapping or covering portions of the live application, as opposed to separate help 
rectangles with help messages. The Guide Character is the representation of the source of 
information and help functions. The Guide Character adds a friendlier nature to the help 
experience and eases the user into the complexities of the Host Application program. 

25 Furthermore, the Guide Character can be an abstract representation and is not required to be 
only a visual or animated character. In addition for illustration, items 435 are animated 
objects that are superimposed on top of the application's GUI interface and are used to 
convey some concept such as in this example the data being loaded into program memory. 
An additional expression of an InfoObj, FIG. 4 further demonstrates the substitution or 

30 complementation of text objects 440 for sound objects as used and presented by the character 
415. The presentation behavior of the text uses numerous techniques that range from static 
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text with scrolling controls to scrolling in any different axis to animated effects. Animated 
text along with display controls are important to handle text that occupies a display area 
larger than the initial window. Additionally the benefits of text is its relatively small data size 
that is vital to networks with slow transfer rates. 

5 Another example of a further variation for CHA Applications, the CHA 

system may also operate in a "Recall Activity for Learning" procedure, which is a CHA 
sequence with additional interactive properties. In reference to the Guide Sequence as 
described previously in FIG. 4, the CHA Sequence has demonstrated to the user where the 
correct interface objects are, and has presented a sequence of steps. Upon completion of the 

10 Guide Sequence, similar to the game "Simon Says", now the CHA system simultaneously 
displays a character 415 on the interface, and asks the user, through the character 415, 
"what's next?" and makes comments, which may be visual and/or aural, as the user inputs 
each step. Intervals of time are measured, and if the user does not respond within a certain 
amount of time, hints from the CHA system are given to the user. This provides positive 

15 reinforcement and active learning. As the user reaches the end of the sequence, the character 
415 may say "you're almost there" or "just one more step". Then "good!" may be stated by 
the character 415, upon completion of the user operations. The character may be given 
different personalities. 

FIG. 5 illustrates the use of an Adjunct Window 850 that coexists and is 

20 displayed with the Host Application. Besides the Host App's GuiObjs e.g. menu items 825, 
shown also with Animated Highlighting 805 and the Guide Character 815, the Adjunct 
Window 850 is a general purpose view area for providing different functionality from 
Activities which are interactive applications, to network activities such as online chat. The 
CHA Activity invoked in the CHA Activity Mode is a complex and robust multimedia 

25 expression and information presentation that is intended to complement, illustrate, or clarify 
underlying concepts behind the operations being demonstrated on the Host Application. 
Running on its own process, the Adjunct i.e. Activity Window 850 provides an independent 
method for handling any type of data display in any format such as graphics, video, and text. 
For example, the Adjunct Window 850 may be used to present an overview map of the entire • 

30 Host App's interface to display in context where the user is currently interacting. 

Furthermore, the Adjunct window 850 can also be used for providing high speed interactive 
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applications serviced by the CHA System's IVM Object using material relevant to the 
operation being described in the Host Application by the Guide Character 815. Events 
occurring in both the Host Application as well as in the Adjunct Window 850 are not 
necessarily independent, or the Host Application and the Adjunct Window 850 can be closely 

5 synchronized through the CHA System's internal messaging. Messages can be sent from 
either the Host Application to the Adjunct Window Application or vice versa. Handling 
procedures can receive messages in either direction and take appropriate action. 

The Adjunct Window is different from an activity window describe herein. 
The Adjunct Window 850 is an auxiliary and complementary part of the operations in the 

10 Guide Mode 305 for use with the assisting Guide Character 815. The graphic representation 
810 may be an image file being displayed by the Host Application which, in this example, is 
a graphics processing program. The CHA Command Objs 820 are optional interfaces that 
provide a controlling functionality for the Guide Sequence being processed such as pause, 
exit, and resume. 

15 In conjunction with the use of interface objects to highlight synchronization 

with a multitasking engine, CHA sequences involving multiple actuations, such as user inputs 
through GUI icons and buttons. If multiple buttons are involved in a CHA sequence, the 
Guide mode 305 can illustrate all of such buttons simultaneously and can maintain necessary 
states from step-to-step. In addition, "Scanning" Interface Objs in the CHA Sequence may be 

20 performed, such that the CHA system quickly "scans" the icons or other actuatable regions on 
the display involved in the sequence in a "gestural overview". For example, for three icons 
involved in a given CHA sequence, the three icons are highlighted on the display quickly in 
the order in which such icons are to be used to give an overview of the procedure to the user. 

The CHA system also illustrates auxiliary processes, by providing animated 

25 illustrations as opposed to static text, which allows the CHA system to provide the expression 
of far more complex graphical interface or structural relationships, such as illustrating other 
alternatives or possible branches of the Host Application program. In addition, as highlights 
are executed, an additional window may be provided which displays animations that illustrate 
different stages or internal data structures or states that reflect how operations are perfoimed 

30 by a Host App that may not be obvious to the user, such that a frame can include animations, 
or provide an interactive interface for describing layers of information that run concurrently 
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as buttons are highlighted or as a user actuates a button or icon; for example, if the user 
presses a button to print hardcopy information through a printer. Although many uses of the 
CHA system may include simple illustrations of the Host Application program, the CHA 
system may also assist to demonstrate applications that are meant to accomplish specialized 
5 or custom features, as well as illustrations of processes as such processes interact with 
interface objects, so such demonstrated applications and processes can be powerful learning 
tools. While known assistance systems may emphasize text to inform a user, the CHA 
System includes both text and multitasking multimedia expression along with optional 
network connectivity. 

10 The Adjunct window 850 shown in FIG. 5 may be displayed through the GUI 

as though mounted on a rotating three-dimensional (3D) object, capable of rotating in 
response to each CHA Sequence Step (SeqStep), which provides strong interface, help, and 
identity of style of the CHA system. Some enhancements of the presentation of InfoObjs 
include a step-by-step approach to assist users. In the prior art, help systems may take a step- 

15 by-step approach and present all steps to the user as an enumerated list. CHA implements the 
steps and also presents the steps to the user one step at a time to reduce screen clutter and to 
allow the user to focus on a single step at a time. As the user interacts with the actual 
interface, steps of the CHA Sequence or procedure are executed. In addition, more than one 
CHA event can take place for multiple interactions with multiple interface objects. 

20 Using the Expression Engine, superimposed graphical effects can also be used 

to emphasize elements of the interface. For example GuiObjs displayed on a screen which 
are not part of a CHA Sequence can be "diffused", that is, a mask bitmap is laid over the 
entire interface with regions in the mask as windows or "holes" for enabled GuiObjs to be 
displayed therein. The mask bitmap is "grayed" out i.e. defocused or colored differently to 

25 simplify the correct interface for the user, and overlaid the interface of the live Host 
Application. 

FIG. 6 shows the basic Activity window 620 in relation to the Host 
Application 605 and its GuiObjs e.g. 610, containing CHA Controls 625 along with the 
character 630 who coordinates events between the Host App and the Activity Window. CHA 
30 Controls 625 contain interfaces for main control of the Activity Window and its general 
operation and processing such as termination of the Activity, closing of the Activity, and so 
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on. The purpose of the CHA Activity Window is to present a user with an interactive 
program i.e. Activity which illustrates a concept or aspect about the Host Application or 
provides training. Although appearing in a separate window and possibly internally in its 
own operating system process, this Activity program is still linked for bi-directional 

5 communication with the Host Application. 

In the given CHA Activity example for the Host App it is the intention to 
teach the user about the location and relation of different commands that are accessible via 
the GUI interface by asking questions. The presentation of activity can be thought of as a 
multiple choice question except that the response controls are animated instead of being static 

10 yes/no check boxes. For example, the character 630 first asks the user a question via voice or 
text such as 635 "How do you load graphic animation files?". The given activity uses the 
CHA System's Expression Engine that multitasks and animates 645 a set of N graphical 
objects 640 i.e. star shapes, where each star represents one GuiObj and responds to user 
mouse clicks. In this scenario, as the objects move around the Activity Window, with the 

15 pointing device, the user interacts with any of the star objects with the intent to reveal its 
associated GuiObj label. As this event is generated through this interaction, a GuiObj text 
name 650 appears. 

The user continues revealing GuiObj text names until the correct response to 
the initial question is found. With a correct response at this point, a visual cue is revealed i.e. 

20 the dart in the target 655 as well as a text message 660. The star shapes 640 stop their 

animation. Concurrently as the target is drawn, a message is sent to the Host Application and 
through the CHA System's application control mechanisms, the appropriate menu item 665 
and related GuiObj e.g. toolbar 670 is opened. The character 630 may also respond with 
positive reinforcement expressed as digitized speech or text. The benefit of the activity is to 

25 provide animated effects and other multimedia processes to create interactive programs that 
help to reinforce important concepts and learning about an application. This simple example 
illustrates one of endless varieties of interactivity that complements the traditional help in 
applications. Programmability of the interaction is completely flexible within the limitations 
of the Expression Engine and its ability to communicate with the main run time components 

30 of the CHA System. 



16 



WO 00/68769 



PCT/USOO/12106 



The CHA system may be considered or deemed, or may be incorporated as, an 
extension to the operating system in that the CHA system provides an intelligent layer 
between the user's input and the functionality and interface presented to the user via the Host 
Application and additional interfaces provided by the CHA system. The function of the CHA 
5 system as an extension may be a platform from which specialized operations may be 
customized by third-party developers to greatly simplify operation of an application to 
achieve useful functionality. Multitasking user interface overlays and InfoObj overlays may 
also be used. 

Customized application hook-interfaces and subclassing components are used 

10 in the operating system with a live Host Application program without access to the internal 
code of the application program, except through public OS interfaces, for example, the 
"MICROSOFT WINDOWS" Common Object Model (COM) object interface technology 
supporting, for example, browser internal component e.g. COM objects. Through the Host 
Application GUI drivers (App GuiDrivers), the CHA system operates to allow for customized 

15 enhancements or facilitators for more efficiently or accurately setting the hooks into the host 
application program for identifying or analyzing run-time user input for a given host 
application which provides for any exceptions to the general or released versions of the CHA 
run-time engines. Interface drivers are also provided as GuiObj handling routines for 
different classes and types of GuiObjs. These handling routines are used to control a given 

20 Host Application. Such App GuiDrivers include algorithms and handler routines for the 
detection and mapping to the logical version of all GuiObjs in a Host Application program. 

FIG. 6A shows an ICAP Map Tool 720 that provides an interface 730 to a 
symbolic representation of the Host App's 705 GUI interface. This symbolic representation 
is linked to the "physical" GUI interface of the Host Application such that selections or 

25 actions on the symbolic version can result in correlating actions on the actual application. In 
this example, the selection of the item FileX 735 triggers an internal message that invokes a 
process that results in the opening of the dialog box "Load animation sequence from file" 71 5 
via actions on the menu 710. The panel of CHA controls 725 provide commands that invoke 
different processes of the ICAP Map Tool and different controls for various CHA System 

30 operational methods that influence the Host Application. 
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FIG. 7 illustrates the major run-time CHA components included in the CHA 
system 1403. With the Host Application 1409 running and ready to be setup for CHA 
operation, on initialization, the CHA inject program 1412 executes and initializes all CHA 
components serviced by their own respective program threads 1406 and 1427. 

5 The user action interception function may involve "MICROSOFT 

WINDOWS"-based subclassing and uses Hooking Functions such as SetWindowsHook, 
UnhookWindowsHook, and CallNextHookEx, and is responsible for intercepting all 
operating system (OS) messages before and after such messages are processed by the Host 
Application. The Hooking Function may be different in targeting various categories of Host 

10 Applications. For example, if the Host App is a stand-alone application such as a word 
processor or a spreadsheet program, then hooking may be performed on the "WIN32" API 
level where message interception is performed on a relatively low level. On the other hand, if 
the target application is a web browser, then the interception function may be performed with 
the detection and analysis of messages on an object level as defined by the HTML Document 

15 Object Model in "MICROSOFT INTERNET EXPLORER." 

Other applications are achievable with a similar model. Because the disclosed 
CHA system includes Hooking Functionality, adaptation to new applications or even 
different operating systems is relatively straightforward with many shared resources across 
the different platforms. 

20 The ChaMap Events module 1418 filters out the many operating system 

messages received, maps the referenced OS objects to GuiObjs in ChaDB, and translates the 
messages into logical CHA Events which are handled by the ChaSys component 1448, which 
acts as a central coordinating unit for the CHA system. The logical mapping of the OS 
messages and objects enables the adoption of the CHA System to any OS or application since 

25 the OS dependent messages are encapsulated in a single Translator Unit for convenient 
mapping. Module IVMinit 1439 initializes the IVMCore 1445, which is the main service 
component for the multimedia expression of any event such as the beginning of an animation, 
a character interaction, spoken instructions, the provision of extensive interactivity, etc. 

ChaSys has a logical view of the Guide Sequence, including the GuiObj view 

30 i.e. all GuiObjs that must be present during the time interval of the InfoObj presentation, and 
maintains synchronization with user actions on GuiObjs with the presentation of InfoObjs. 
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ChaSys has the ability to suspend or interrupt the InfoObj Services Module i.e. the process in 
the Expression Engine, and to move the Sequence Step (SeqStep) either forwards or 
backwards. The Guide Sequence processing may make adjustments midstream, if necessary, 
so either User Actions are controlled or the Guide Sequence and Expression Engine are 
5 adjusted. 

During run-time, the CHA system accesses the ChaDB 1451, which is the 
CHA Database of collected data related to the Host Application that is prepared prior to 
operation of the CHA system, with the ChaDB being in the CHA format and consistent with 
the Host Application Model for efficient run-time operation and data preparation. The GUI 

10 Analysis Resources (GUIana Res) component 1454 part of the CHA Development 

Environment is used during the CHA application production process, with the primary goal 
of the GUIana Res 1454 being to collect pertinent information via a SpyMap component 1424 
during a learning mode for learning the operation of the Host Application, and to prepare 
CHA data formats used during run-time. 

15 The Event Queue 1422 is established to asynchronously collect and transmit 

messages from the Host Application to ChaMap 1418 which passes filtered and processed 
data to the ChaDB module 1451 as well as CHA Events to ChaSys 1448. ChaSys operates as 
a central managing coordinator of numerous processes. 

In "MICROSOFT WINDOWS", subclassing is a mechanism which is 

20 provided to allow for the establishment of an intercepting routine or function to monitor and 
process messages before or after such messages are passed on to the Host Application. Such 
interception functions permit control of the behavior of the Host Application and the adding 
of additional functionality which may not have originally existed in the Host App, such as the 
invoking of new functions, the disabling Host App GuiObjs, etc. 

25 Referring to FIG. 8, the Host Application has been launched in step 1 505 and 

is running in the operating system. Part of the basic system of the Host Application is a 
process allocated by the operating system and one or more running threads such that the 
process handles the multitasking within the Host Application program. 

On initial launch in step 1510, the CHA system loads its critical components 

30 into memory and also establishes links with existing operating system services to prepare to 
perform the CHA processes. Initialization options are accessed in step 1515 from an external 
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file e.g. .ini file or other data registry which is optionally in ASCII or binary form, and the 
CHA system sets up various initialization parameters such as a file path for CHA System 
components, communication parameters such as network information, as well as information 
about the target Host Application such as window class structure identifiers which aid in the 
5 identification ofHostApp windows and other GUI objects, etc. After the Initialization 
Options are read, the operating system global data repository such as an operating system 
Registry, is read and additional CHA program values, such as directory paths, general system 
capabilities, and/or the CHA program identifying key names are fetched and stored for later 
reference. 

10 An additional method of passing parameters to the program is performed via 

the command line which is a convenience for launching the program in different Main 
Modes, for example, for running the CHA application in a debug mode to assist the 
development process. The initialization process then performs a quick look into the operating 
system to seek and locate the main window handle for the target Host Application. The 

15 handle allows all existing windows to be enumerated, and a match is sought for the key 
window identifier found at the end of initialization in step 1515. In many modern operating 
systems, the dynamic link library (DLL) or its equivalent is typically employed which 
provides the capability of automatically loading and running external re-entrants, that is, 
shareable code in an already running process. In such situations, the CHA System first 

20 attaches itself to the Host Application which is already running in the operating system via a 
special function, for example, the LoadLibrary function in "MICROSOFT WINDOWS", for 
finding and loading the specified DLL file in step 1520. 

Thread determination specifies whether the DLL runs in its own thread or not, 
and such thread determination has shared memory implications. The main CHA.DLL file is 

25 loaded, after which the OS generates a message, for example, a DLL_PROCESS_ATTACH 
message. Upon the receipt of this message indicating when the DLL "attaches" itself to the 
process, pointers to callback functions in step 1525 are setup which are mainly functions for 
receiving, filtering, and pre-processing messages before they are sent to the Host Application 
1505. 

30 An Interactive Virtual Machine (TVM) is then loaded or invoked in step 1527 

which is the primary multimedia services engine which acts as an Expression Engine for 
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many different graphical events that are synchronized with sound, and which also enables 
high performance interactivity. On receipt of the initial create message, for example, 
MPHJVM_CREATE, a test is performed in step 1530 to see if the IVM object is already 
loaded, and the CHA system proceeds to initialize the IVM object in step 1 535 where an IVM 
5 thread is created to enable IVM code processing. During the initialization stage, memory is 
also allocated in step 1540 for general purpose use in the IVM engine, with such memory also 
known as aheap. 

Communication between different "MICROSOFT WINDOWS" OS processes 
or threads and the creation of synchronization objects can be implemented in a number of 

10 ways from and including shared global system memory queues, mutexes and critical sections 
to Windows Message Queues. In the first i.e. critical section technique, for additional 
connectivity to the Host Application, the API function InitializeCriticalSectionO in the OS is 
called in step 1545 which establishes the ability to pass data safely between different threads 
and to synchronize between many different messaging events between various CHA 

15 components and the Host Application. 

Once the MPH IVM CREATE message has been processed and an instance 
of the IVM object and thread have been created, step 1550 is then performed in which 
additional hook information is then passed to the next procedure in the "hook-chain" which is 
one of a series of callback functions called at intervals, depending on user input and OS 

20 determination. Still another implementation method for multiple processes or thread 

communication and synchronization is to utilize OS message queues e.g. the MICROSOFT 
Windows Message Queues. Windows Message Queues are the primary mechanism used for 
communication between modules of CHA system running on different threads, such as 
ChaSys and ChaHook. The synchronous mode of that mechanism, via a SendMessageO API, 

25 is also used for synchronized access to some shared resources within CHA without 
employing "MICROSOFT WINDOWS" specific synchronization objects such as critical 
sections or mutexes. Windows Message Queue is also used for communication between 
modules running on the same thread, for example, ChaSys and IVM, for effective separation 
of the modules as well as easier implementation on platforms other than MS Windows. 

30 As the CHA System's Expression Engine, the IVM Class includes a number of 

public and private functions that service the expression, which may involve any number of 
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events, from one event to N events. The inner workings of this engine are very extensive and 
cover a very broad range of applications. Essentially a multitasking language and processor, 
the IVM provides the backbone of all multimedia services from simple GuiObj interface 
highlights to independent, complex, full featured interactive applications. Access to the very 
5 powerful multimedia services is handled by several categories of interfaces as follows: 
System, Control, Status, Media Services, Messaging, Networking, and input/output. 

The System interface includes, among others, the following functions: 

Initialize, Start, Pause, ShutDown, Resume; 
the Control interface includes the following functions: 
10 Synch Operations, AcknowledgeReceiptMsg, Notify, CurrentValue; 

the Media Services interface includes the following functions: 

Task Controls, Proceed, SendVars, ReceiveVars, OnReceipt, 
PlaySequence, CreateTask, KillTask, KillGroupTask, CreateGroupTask, RefreshBackground; 
and 

15 the I/O interface includes the following functions: 

FileRequest, FileLoaded, FileReady, FileStartReceiving, 

FileBlockStatus. 

The Host App Model provides a fundamental data set for functions and tools 
to provide many automated operations for dynamic or automated ICAP and Guide Sequence 

20 construction. The Host App Model includes a structure that represents every GuiObj in the 
Host Application stored within structured, hierarchical and graph formations. The Host App 
Model also includes data structures and link information that reflect the entire user interface 
of the Host Application as well as GuiObj details and interface groups and their behaviors 
and relationships. While the Host App GuiObj records have all instances and different 

25 categories of GuiObjs and their states for the Guide Sequence Set or functionality, a Link Set 
of the Host App Model also helps track the state of the Host Application and stores all 
description of the interconnection between the GuiObjs. By using heuristics on the Host App 
Model's data and descriptions, many syllogistic, searching, predictive, and locating 
operations can be executed to create inference or deductive behaviors in conjunction with the 

30 user's actions within the physical interface for a Host Application. 
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One of the functions of the Host App Model is to facilitate rapid access to 
perform construction and real-time assembly of CHA Fixed and Dynamic Guide Sequences 
during the Host App program session. Such real time assembly of sequences differs from 
known assistance systems, since known assistance systems instead embed hard coded scripts 

5 into a help application. Real-time assembly and Guide Sequence generation facilitates a 
much more dynamic interaction with an application such as performed and experienced in the 
CHA System's Explore Mode. Dynamic path finding functions can find alternate routes to 
reach a destination, and parameters such as controlling the number of steps needed to reach a 
destination i.e. particular GuiObj, estimated data size, etc. are also input. The path finding 

10 functions then calculate the route through the Host App Model's database. Encoded 
structures can be embedded into the database to height" certain pathways over others to 
further control the heuristic engines. 

The Host App Model structure also provides a platform to perform analysis for 
CHA Monitoring operations such as contextual Inference Processing utilizing the CHA 

15 Inference Engine. Via natural language or list processing, if some intent is communicated to 
the CHA System such as a more specific target or general description such as font controls in 
a word processor, the CHA System's Inference Processor can suggest subsequent or alternate 
actions. The user may be prompted for such intent. Alternatively, if the process is automatic 
without prompting the user for an indicated intent, the CHA Monitoring mechanism can also 

20 attempt to present alternatives based purely on calculations performed by the Inference 
Engine in collaboration with the Host App Model. 

The user's intent can also be input through an Assistance Topic which 
describes the identification, classification or categorization of a given Target GuiObj Action 
Set and processes to an associated lookup key. For example, a Target GuiObj Action Set may 

25 include using a "Tab" button control in a word processing program in which various tab 
related controls reside in different locations in the Host App's interface. 

Another application of the Host App Model is to assist the automatic 
construction of ICAP commands. With an ICAP command, to change one or more Host App 
parameters e.g. set by a checkbox control in a dialog box, the CHA ICAP automatically 

30 executes a series of program actions without requiring any user interaction. This is 
accomplished with CHA Mapping functions in which GuiObj Paths through the Interface 
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Hierarchy can be mapped to actual "physical" controls in the application. Similarly, a user's 
interaction with a physical control in the application can map to the logical representation of 
that same control i.e. the GuiObj within the Host App Model. 

The CHA's Host App Model and associated Assistance files can be built either 

5 during the development of the Host Application with general access to the Host Application's 
internal source code files, or after completion of development of the Host Application. In the 
case of post-completion access, a third-party developer can develop the CHA Assistance 
system and method after the commercial or internal company release of a product or 
application. This independence permits third-party development for Host Application 

10 support in very specific areas of the application without requiring access to the Host App's 
original source code or libraries. With the publication of the specifications of the Host App 
Model, third-party developers are able to develop their own equivalent CHA Systems. 

The CHA System's ChaDB file set is the primary data and code storage area 
for CHA Assistance support and execution of Fixed and Dynamic Guide Sequences and auto 

15 generated ICAP Procedures. The ChaDB file set consists mainly of the following set 
declarations: 

Host App ChaDB = Host App Model + Host App Model Resources 

+ Host App Assistance Set; 
Host App Model = GuiObj Set + GuiObj Link Set; 
20 Host App Model Resources = Commentary Library + Behavior Library 

+ Miscellaneous Libraries; 
Host App Assistance Set = Assistance InfoObj Set + Assistance Script Set 

+ User Custom Data. 
GuiObj Set: persistent representations of all interface objects in the Host 

25 Application 

GuiObj Link Set: all link information between all GuiObjs 
Commentary Library: component material for assembling commentary that is 
interleaved into Guide Sequences. 

Behavior Library: a set of available behaviors described in Script/language 
30 Opcodes that refer to the environment resources that defines a controlled set of interactive 
operations with the Host Application and CHA events. This permits the substitution of data 
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elements and the reuse of Behavior code without the need for reprogramming to produce new 
Assistance Sets. 

Miscellaneous Libraries: contains shared resources or code for various general 
purpose functions. 

5 Assistance InfoObj Set: the essential information in any CHA compatible 

format that is used in Guide Sequences. 

Assistance Script Set: script or language opcodes that describe Fixed Guide 
Sequence events 

User Custom Data: holds user customized data in any format to allow 

10 personalization of CHA applications. 

In general the database is organized using a graph structure. While there exist 
hierarchical forms and groups of GuiObjs that reflect a typical Host Application's natural 
internal organization, there are, in addition, links between GuiObjs that allow closer 
connections along strategic navigational pathways e.g. returning to a root position. 

15 Connectivity algorithms enable shortest path routes to be determined. Additionally a graph 
structure provides multiple Relational Views from the same database necessary for the 
different functionality. For example, one view may reflect the hierarchical structure of a Host 
Application's interface. Another view may reflect the operating system's organization, while 
another view may be navigational, etc. 

20 Referring to FIG. 9, a chart is illustrated which describes the major data 

partitions of the CHA Database. The Host App's CHA Database (ChaDB) is divided into two 
major divisions: the Host App Model and Libraries 1602 and the Assistance Object Code and 
Data 1604. The Host App Model and Libraries include the Host App Model 1603 and several 
auxiliary libraries, such as a Behavior library 1606, a Commentary library 1609, and 

25 Miscellaneous Libraries 1612. 

Each of the libraries represents shared data or code which provide a set of 
resources which can be referenced by the Assistance Objects Code or which can be processed 
during run-time to produce combinations or permutations of results that can be packaged and 
accessed by the CHA system during run-time to assemble Guide Sequence Elements. The 

30 Assistance Objects represent CHA application content which include both code 1615 and 
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resource data elements 1618. Additionally there is a User Custom Data Set 1621 which 
includes user-inputted, customized data. 

FIG. 9 also illustrates a ChaSys Script object, including components 1666- 
1675, and additionally, the Host App Model including components 1622-1663. The ChaSys 
5 Script N object includes elements representing the code 1666 and data 1669-1675. As shown 
in FIG. 9, the code and data represent the code references to the global Behavior Library 1606 
which includes code descriptions of generic behaviors for prescribed GuiObj interface 
configurations and environments. The data elements of the Script structure are referenced to 
areas of the Media InfoObj Resources 161 8 or User Custom Data 1 621 . 

10 The Media InfoObj Resources are further subdivided into multiple Detail 

Levels for access at different levels of information from summary information to detailed 
information. The Host App Model is represented as a graph structure including GuiObjs 
1627, 1639, 1651, and 1660 which are indirectly linked together by GuiObj Link Structures 
1633, 1645, and 1657. Furthermore, individualized InfoObjs 1624, 1630, 1636, 1648, 1654, 

15 and 1663 are connected to all of the GuiObj and Link structures. 

There are many possible views of the data and the infrastructure of a Host 
Application's interface. One illustrative embodiment is shown in FIG. 10, which shows the 
ability of the CHA system and method to traverse any GuiObj Path. As shown in FIG. 10, an 
example of a series of user steps through a GUI style program interface is shown which 

20 navigates through a mixture of menus, menu items, and dialog boxes to reach a Target 

GuiObj 1726. Beginning at the general default state of the Host Application which normally 
is at its main window with access to the current loaded file being edited, the location 1702 has 
a group of Menu "heading". When a Menu heading is actuated with a pointing device or 
other input devices, such as the MenuB selection 1 704, the CHA system invokes a group of 

25 menu items as shown in the menu list 1706. The user GuiObj itemA in turn invokes the 
dialog box DlgA 1708 which includes numerous other GuiObjs, of which the Cancel 1710 
and TranspA 1712 GuiObjs are designated as Transport GuiObjs. 

In this context, a Transport GuiObj is a type of GuiObj that navigates the user 
from one GuiObj Set to another GuiObj Set, with a group of GuiObjs including a particular 

30 GuiObj Set, such as Dialog Box A (DlgA) 1708 and Dialog Box B (DlgB) 1714. Transport 
GuiObjs are used to navigate through the application. Note that in a simulated mode of 
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operation, Highlights are shown for the relevant controls that could execute the goal task but 
are not actually activated. This simulation mode may be used for situations in which one 
wishes to show the user where the needed controls are located but does not want to tamper 
with the user's environment. 

5 Each GuiObj Set is part of the Host Application state in the context of the 

target application, which a user navigates from one dialog box to another, which may be 
referred to as having the user move from one Host Program state to another Host Program 
state. The button interfaces 1 709 and also in the remaining dialog boxes 1715 and 1723 are 
additional GuiObjs in different categories and different Host Application states. 

1 o In a subsequent step, the user actuates the TranspB choice 1718, which 

invokes the group of menu items Iteml, Item2, Item3, etc. which is another GUI GuiObj type. 
Clicking on Iteml in turn brings up the next dialog box 1722 which has a slightly different 
style in the dialog box 1 722, and is grouped with other dialog boxes that are accessed via Tab 
GuiObjs 1724. From the dialog box 1722, the button C7 1726, which is a Parameter GuiObj 

15 type, sets a single parameter and affects the Host Application program's persistent internal 
data settings which in turn applies to the process of calculation or display and functionality. 
The Cancel Buttons 1710, 1716, 1728 alter the Host App's state from the current set of 
GuiObjs to the previous state until the program once more reaches its general default 
condition 1702 where editing or processing access is again available to perform operations on 

20 the current file data. This default condition is also referred to as the Main Ref State. 

There are several types of GuiObj data structures shown in FIG. 10, such as 
Nodes, Transport, Cancel, and Parameter Selection. As shown in this example, a series of 
GUI interfaces of a specific type are provided, such as Menus 1702, Menu Items 1706, Tabs 
1724, and Dialog Boxes 1708. There exist an endless variety of combinations of the many 

25 different types of GuiObjs. Many other combinations using many different types of GuiObjs 
are possible. The principles described, however, are the same for any of the many GuiObj 
types as related to the disclosed CHA system and method. 

The Main Menu GuiObj Node 1730 describes and includes data relevant to 
each of the Main Menu items, and the data structures of FIG. 1 0 illustrate a high level conduit 

30 network between all Host Application states and between all navigational departure and 
destination points. Each Transport GuiObj data structure 1732, 1734 includes relevant 
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linking information to other peer Transport GuiObjs as well as a single link to another 
GuiObj Group corresponding to a GUI object such as a dialog box or menu group. 

Every GuiObj Node has a key value determined during the CHA development 
phase and Host App analysis using the Guidance Development Tool Set. The key value is 

5 used to help locate a particular Node data record. The next Node includes data for the 
GuiObjSet which is a DropDown Menu 1706. Although the GuiObjSet includes a set of 
Transport GuiObjs to link to child structures below itself, the GuiObjSet does not include a 
Cancel structure because the behavior of a DropDown Menu is to disappear when a user 
releases the left mouse button, or an equivalent actuating action, and hence pointers bypass 

10 this record when moving up the record hierarchy. 

The structure 1738 is the first Dialog Box encountered by the user in this 
example and includes a set of structures 1744 associated with additional user GuiObjs 
included in the Dialog Box. As the Target GuiObj 1726 and its associative data structure 
1762 is the goal, structures 1738 are skipped. The current Transport GuiObj 1 740 leads to the 

15 next Node structure which is another Dialog Box which also includes its own Parameter 
GuiObjs 1748. Links continue through at least one additional DropMenu structure 1754 until 
finally arriving to the Dialog Box which contains the Target GuiObj 1762 found in the 
dialog's GuiObj structures 1760. 

Every GuiObj can have an associated InfoObj which includes all data that 

20 presents some form of information in any of a variety of formats. The goals of the data 
InfoObj Data Structure are to facilitate rapid access, data integrity, and ease of maintenance, 
and also to provide a facility or tool for performing InfoObj Detail Level assembly or 
retrieval. InfoObj Detail Levels are ranges of detail that can be associated with different 
categories and groupings of GuiObjs. One category is the Group Level which contain 

25 summaries for groups of controls such as those controls included in a dialog box. Another 
category is the Control Level which reveals specific information about a control e.g. a 
checkbox, what the control does and what the control controls. Another category is the Topic 
Summary Level which includes summary information for a general category e.g. formatting. 
Furthermore, the InfoObj Detail Level (IODL) facilitates further divisions of detail 

30 determined by the CHA developer, which ranges from high level, summary to deeper 
specifics. Data structures related to IODL facilitates runtime processing and selection of 

28 



WO 00/68769 



PCT/US00/12106 



appropriate InfoObj records. Special Notes include miscellaneous prominent points about 
GuiObj controls contexts or specific tips, etc. 

With reference to the Node structures shown in FIG. 10, such as structure 
1 730, FIG. 1 1 is a detailed illustration of a Main Menu Node's associative structures 1 805 in 

5 which each Transport GuiObj 's attached InfoObj 1 8 1 0 is further expanded into the sub- 
structures contained in 1815. The Control Data 1835 enumerates internal InfoObj details, 
such as metrics and relative sizes, offsets, pointers to external records, and other control 
information about its linked category structures, such as: InfoObj Syntaxes 1820, Sequence 
Media 1 825, and Expansion 1 830. The Syntaxes category may contain records related to the 

10 InfoObj Detail Level parameter that influences run-time record selection in Dynamic Guide 
Sequences: Initial Explanation which associates via pointers, summary or high level content 
describing the associated GuiObj; Transport which associates via pointers, content attached to 
Transport GuiObj types to provide traversal through the Host App's interface hierarchy; and 
Destination which associates via pointers, content describing a Target GuiObj, with all Host 

15 App parameter changing GuiObj controls could have Destination content record to facilitate 
satisfactory Dynamic Guide Sequence generation that expresses the conclusion of a series of 
SeqSteps. 

The InfoObj Media category may contain records of different media types: 
Programming Sequences which is code that can program sets of events that can be referenced 

20 externally to promote modular design structure, which may be related to Expression Engine 
execution or to ChaSys code references; Text Info which is any graphical expression of text 
in static or animated form; GraphicElements which is any pictorial representation static or 
animated rendered by the Expression Engine; Audio Info which is any digitized or control 
information related to aural mechanisms; InterActiveContent which is any format used by an 

25 Expression Engine, as exemplified in the CHA Activity in FIG. 6, which enables ftdl 

interactive behavior e.g. the control of a multitude of animated objects with a pointing device; 
System ControlCodes which is additional control information to facilitate efficient interface 
and operation by CHA engine subsystems; Comments which is a data record to store user 
custom information that is related to the previously described data structures. The last 

30 category depicted i.e. Expansion 1 830, is a mechanism for expanding existing CHA data files 
without disturbing legacy data sets. Note that in terms of the kind of relationships between 
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the categories, each instance contained in the Syntax category may have 1 or more associated 
instances contained in the Media category to support the Detail Level. The GuiObj records 
within the Host Application Model have pointers connected to InfoObjs of many different 
data types in different formats for a wide variety of expressions from text to static graphics to 

5 animated or interactive applications. 

Referring again to FIG. 10, for an example of an application of this Host App 
Model, consider the problem of Path Tracing through the model to find routes between 
GuiObjs and the subsequent automatic generation of a Guide Sequence. 

In this Path Tracing mode, initialization includes the defining of a Start Point, 

10 which could be the Main Ref State or alternately, a different starting GuiObj location selected 
by the user via a displayed symbolic representation of the Host App Model. The CHA 
system and method then fetches a Behavior Script from the Behavior Library, or otherwise 
generates an appropriate behavior script, and then retrieves the initial data from the Host App 
Model. The user may interact with the actual Host Application and locate a specific target by 

15 placing the cursor over the Target GuiObj. As the user confirms with an action the Target 
GuiObj, operating system data is extracted and stored i.e. as a Target GuiObj Structure. The 
extracted internal OS data is converted to a key value which is sent to the Host App Model 
Processor 2040 shown in FIG. 12 at which cFindGuiObj or cFindPath functions may be 
executed. The user may also click on the exit Path Trace control, and traversal of the Host 

20 App Model ends. 

Before proceeding the user can select one of the following settings for the 
interaction functions of the CHA system and method: a Verbose Flag or Disable Verbose 
Flag, and 

a InfoObj Detail Level value. These settings determine the levels of associative information 
25 via InfoObjs that are presented and interleaved during the generated Paths. 

Continuing, to perform a Path Trace Phase, the CHA system and method start 
from a Target GuiObj location, and traverses back through the hierarchy of GuiObjs to locate 
the Start Point. Each Transport GuiObj Struct is then stored in a buffer as each Transport 
GuiObj Struct is located, and each instance of such Transport GuiObj Structs is tracked; that 
30' is, a SeqStep structure, index, and set up links are defined between each SeqStep. Starting 
with the Target GuiObj, within the current Node Structure, the Path Trace Mechanism locates 
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the immediate parent or parents of the Target GuiObj. Note that the term "parent" refers to 
the parent/child relationship which suggests the containment of one i.e. the child by another 
i.e. the parent. A child may have multiple parents and a parent multiple children. The many 
possible interconnections are a reflection of the Host Application to efficiently and accurately 

5 reflect the application's structure. Continuing, if there is more than one parent, for each 
parent N, the paths are traced until the Start Point is found. If only a single parent is 
involved, the Path Trace Mechanism resumes by locating the new node and locating its 
parent, and the traces are continued until a Start Point is verified. 

If no Start Point is found on the current route, the CHA system and method 

10 pick up the Nth parent alternate route, and repeat the processing until a Start Point is located 
or inferred as an option i.e. the Host App's Main Ref State. If not located, then the CHA 
system displays an error message and resets. The various GuiObjs have attached InfoObjs 
that can be retrieved during Guide Sequence execution at different Levels of Detail 
determined by parameters set earlier. The CHA system also may refer to the Behavior Script 

15 or generates Input Scripts as needed, and extracts script fragments and inserts the fragments 
into the SeqExecBuffer. Note that the SeqExecBuffer is a major "collection" buffer that in 
both Fixed and Dynamic Guide Sequence preparation assembles all required code and data 
elements prior to execution of one or more Guide Sequence steps (SeqStep). Behavior 
Scripts provide executable OpCodes that are pre-compiled or compiled at run time, and are 

20 designed for different window configurations and environments to provide various Guide 
Sequence Behaviors. * 

For each SeqStep generated in the Path Tracing Phase, the CHA system and 
method insert into the SeqExecBuffer at least one of a Script Instance, a Commentary 
Instance, InfoObj Instance, and a Highlight Instance. A Script Instance is a Script code or its 

25 equivalent Opcode Set that is executable by the ChaSys Processor along with pointer 
references to data resources or their equivalent. The series of SeqSteps i.e. the Dynamic 
Guide Sequence is passed into the Sequence Processor Modules to complete execution. 

As explained previously, a Sequence is a term used to describe the sequential, 
step-by-step presentation of information via InfoObjs synchronized with the user's 

30 interactions with a Host App's GuiObjs. GuiObjs can be controlled by the user or by a CHA 
process using its own independently generated schedule of events in which the user is 
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prompted to act or provide information. A Sequence includes N SeqSteps in which each 
SeqStep has zero or more associated InfoObjs or defines the interaction with one or more 
Host App GuiObjs. As previously mentioned, InfoObjs have various presentation forms 
ranging from static or animated graphics to static or animated text. The SeqStep may have 
5 associated InfoObjs that are presented in different expression forms synchronized with user 
input and GuiObj related CHA Events and additionally, the GuiObj 's highlighting 
mechanism may be animated, static, or associated with special graphical effects. 

A possible user experience with a Guide Sequence may involve the situation in 
which the user, after reading the text and graphic or listening to an audio description from an 

10 InfoObj instance, actuates a CHA interface item e.g. a button or icon which launches the 
Guide Sequence. A display status message appears, followed by CHA processing which 
loads the first targeted section of the Host App's GuiObjs Interface that the user may monitor 
and follow. Upon the loading of the Host Web Page or Application GuiObjs, the targeted 
interface elements on the Host App are highlighted and InfoObjs; that is, messages or 

15 interactive elements, are displayed. The user can further interact either with the Host App 
directly or with different CHA controls such as Next, Back, or Random- Jump-To-SeqStep 
controls. Additional SeqSteps are processed until the current Guide Sequence is completed or 
aborted. In the latter case the appropriate status message is displayed and all input control on 
the Host Application can return to the Host Application unless another sequence is selected 

20 from the Search Tool or some other method of invocation. 

Sequences exist in two principal categories: Fixed or Dynamic. A Fixed 
sequence has the basic GuiObj Path which is predetermined, and combines data prepared 
during the CHA development process prior to the interaction with the Host App session. A 
Dynamic sequence does not have a predetermined GuiObj Path, but instead the GuiObj is 

25 selected on demand via user actions and subsequently derived from the Host App Model and 
its heuristics. In both Fixed and Dynamic cases many different elements belonging to the 
Guide Sequence can be actually assembled or constructed during sequence processing in new 
combinations that are unknown during the time the sequence was developed. In other words 
many elements such as Commentary i.e. language phrases expressed from the CHA System, 

30 or different combinations of InfoObj Detail Levels are combined during user actions or 
requests during the session with the Host Application. In general a Fixed Guide Sequence 
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allows for fixed paths and exception processing for creating finely tuned sequences but may 
still use just-in-time generated elements as well. 

A Dynamic Guide Sequence is suitable for rapid access and/or production 
where the output product quality is enhanced by the heuristics of the various element 

5 assembly engines provided for the different data types e.g. Commentary or InfoObjs, with the 
InfoObjs having parameter Detail Level controls. Both Fixed and Dynamic Sequences have 
their place in the CHA System's ability to provide a broad range of assistance quality for 
different solution contexts. 

The CHA System allows for both these Fixed and Dynamic categories to 

10 operate in the same session or environment in which Guide Sequences are either assembled at 
run-time or are prepared before run-time, and the Guide Sequences include all relevant 
InfoObjs to convey information and guidance to the user. The CHA System also permits the 
switching between Fixed and Dynamic operational modes on demand. For all of the 
preceding, although processing for a single application may be primarily performed, multiple 

15 applications can also be controlled and coordinated in the CHA System. 

FIG. 12 illustrates a set of modules that process CHA Guide Sequences. Items 
2005, 2010, and 2015 are possible search tool interfaces that in turn invoke processes leading 
to specific information or access to one or more Guide Sequences. In this assistance mode, 
the illustrated search tool interfaces enable the invocation of one or more Guide Sequences. 

20 Such tools could employ known techniques in the art such as voice control input, standard 
GUI interfaces, natural language parsing or inference engines that map user input 
descriptions, actions or selections to specific output i.e. a CHA Sequence, code instruction 
sets and other types of CHA data forms. The Central Search Module 2020 employs various 
methods to perform a search technique within or by the selected search tool to perform the 

25 lookup function for retrieving specific Sequence instances. 

Depending on the search tool interface or the mechanism selected, one or a 
combination of detection and selection techniques are enabled and used to process and narrow 
the user input to perform assistance retrieval. The data set being searched includes prepared 
assistance data and optionally custom data created by the user via one of the CHA System 

30 procedures such as ICAP. Another configuration and interface technique for locating an 
assistance instance includes using the Monitoring and Inference Module 2015 to 
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accommodate the analysis of the user's actions to make automated inference of the interaction 
with the Host Application. Once detecting specific actions, the derived description or keys 
determined through the analysis are used as input to the Central Search Module 2020 that 
invokes the lookup function in the existing ChaDB data set. 

5 The Sequence Manager Function 2025 is a function that manages the count 

and termination of N retrieved or processed Sequences. The ChaDB Retrieval Module 2030 
performs all data record retrieval from the ChaDB data set which includes all necessary 
components such as the Host App Model, the InfoObj Set, the Commentary Data Set, User 
Custom data, CHA Guide Sequence Script code, etc. Depending on the Host Application 

10 environment, this module also performs data caching, the starting of asynchronous loading 
processes, or compression before transmission, or conversely, the compression of data on 
receipt from a remote location. 

The Data Expansion Module 2035 converts data from its compact storage and 
optionally from a compressed form into the expanded CHA run time format The Host App 

15 Model Processor Module 2040 handles lookup and concurrent navigation/retrieval and instant 
construction of InfoObj data components. The Commentary Assembly Module 2045 does 
rapid sentence and phrase construction for non-essential data type such as presenting various 
comments as the user interacts with the Host Application. These features of the Commentary 
Assembly Module 2045 permit the CHA system and method to create a less repetitive, more 

20 dynamic personality for the Guide Character. Guide Sequence processing also uses the Main 
Sequence Processor 2050 which synchronizes, coordinates and manages different processes 
and sub-processes for Guide Sequence Script commands. Module 2050 contains the ChaSys 
component which plays an important role in coordinating activity and communication 
between different concurrent tasks and processes. 

25 The Procedure Execution Module (PEM), in cooperation and communication 

with the Main Sequence Processor, parses all data structures and types in the SeqExecBuffer 
and launches or routes them to the appropriate service inputs such as in the InfoObject 
Services Module. The PEM also includes the function cStoreExecBuff which parses prepared 
output from different modules and stores instances of the output data records into the 

30 SeqExecBuffer. Furthermore, the processing of these various data types can be performed 
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asynchronously depending on the requirements of the environment, such as in a networked 
environment with slow data transfer rates. 

Both the Host App Subclassing Object 2060 and the CHA Message Mapping 
2065 are modules, also described with reference to components 1415, 1418, & 1421 in FIG. 

5 7. Such modules 2060, 2065 handle the interception of Host Application operating system 
messages and the filtering and mapping to CHA Events which are then passed to the Main 
Sequence Processor 2050 to coordinate and synchronize user input actions on the Host 
Application with internal processes and sub-processes being managed for the presentation of 
InfoObjs. The expression and manifestation of these InfoObjs are handled by the InfoObject 

10 Services Module 2070 which selects the appropriate expression engine for the input data 
types. The appropriate rendering function processes its native data type to manifest aurally or 
visually the InfoObjs to the user. Furthermore, the InfoObj Services Module is a framework 
for embedding different types of InfoObj or Expression services which can be enabled 
depending on different configurations of environments. For example, the IVM class can be 

15 substituted with a different expression engine or perhaps a different rendering mechanism 
such as a text mark-up language similar to HTML. The Exclusion Processing Module 2075 
evaluates exceptions such as incorrect user actions and routes messaging to appropriate 
subsystems to construct and select appropriate responses handled by the Expression services. 
The General Services Module 2080 handles all text, static and animated graphics and aural 

20 expressions for all non InfoObj services such as GuiObj highlighting and overlays, the Guide 
Character, Commentary objects, etc. 

Referring to FIG. 13, during Guide Sequence Initialization 2105, the run-time 
Modules are loaded into memory, such as random access memory (RAM). The run-time 
Modules are then launched, and a primary Host Application is selected. The "primary" 

25 descriptor refers to the designation of a leading i.e. primary application within a group of 
applications in which actions are initialized and managed. Eventually after a number of 
SeqSteps, processing may return to the Primary Application. For example, a web browser 
presents to the user an interface for an information set and is the primary launch point for a 
series of actions on one or more client applications on the user's computer and is also a 

30 potential point of return when all SeqSteps are completed. Another example is when the user 
begins a sequence of operations on a local client application i.e. Host Application, chooses to 
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manipulate settings in their local operating system e.g. the "MICROSOFT WINDOWS" 
Control Panel to change a system setting, and then returns to the starting application. The 
primary application setting is dynamic and can be changed after initial processing, if required. 

During this Initialization stage, the CHA Host App ChaDB or multiple ChaDB 

5 files are set up for processing, and links are established with the CHA Run Time Modules to 
properly locate needed run-time data. This group of multiple applications are called the CHA 
Application Set, which represent all applications being controlled and monitored in relation to 
the Assistance Data Set provided. This accommodates typical modern multiprocessing, 
multitasking operating systems which permit the concurrent running of numerous 

10 applications. The CHA System can operate and coordinate actions as well as processes 
across multiple applications. 

The pointer to the current ChaDB, which had been initialized during the 
application selection phase performed at the Initialization stage 2105, as well as a pointer to 
the ChaDB local index inside ChaDB, are found. The local index permits all internal ChaDB 

15 operations to execute to retrieve the targeted Guide Sequence using the Assistance Item Key. 
The Assistance Item Key value is an identifier used to locate an Assistance record which 
includes a header and set of pointers or functions that enables access to all relevant data for 
the given Guide Sequence. The ChaDB index operates in cooperation with the ChaDB 
Retrieval Module 2030, shown in FIG. 12, required for setting up a mechanism to handle all 

20 resource management during processing of the Guide Sequence. This resource management 
is needed to track all code, InfoObjs or InfoObj elements to determine where the resources are 
located at any given time to schedule delivery. If marked as local, then a request is made to 
the input/output services to load the binary data, to decompress the binary data if necessary, 
and to pass it over to the calling function making the request. If not local, then request 

25 messages are sent to the Resource Locator to begin an asynchronous or semi-synchronous 
process to move data across different boundaries whether these boundaries are across 
applications or network connections of varying speeds. 

There can be any number of searching tools or interfaces that map access of 
any number of Guide Assistance instances from which the user makes a selection or sets an 

30 option that transparently invokes a search method 21 10. For example, a simple search tool 
interface may include a form to fill in text describing the topic of interest whereupon a 
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database search is executed and results displayed in any combination of formats such as text, 
graphics, animations, or a set of Guide Sequence launch controls e.g. icons. The input text 
may be simple string searching or may be processed by a natural language parser to provide a 
user-friendly interface. 

5 Another possible interface includes an Interface Map Tool which presents one 

or more hierarchical tree style views of the Host App interface, its principal GuiObj Groups 
and other secondary or multilevel interface objects in its hierarchical organization. Refer to 
the example shown in the ICAP Control Map illustration in FIG. 6A. Being a symbolic 
representation of the GUI interface of a Host Application, this representation may include, for 

10 example, a tree-structured 2D interactive map or a 3D version. This Interface Map tool 
provides an interactive interface for the user to select one or more GuiObj destinations which 
the CHA System can navigate or which executes a series of actions automatically. From this 
interface the located destinations are input to the CHA modules that can then dynamically 
lead the user through a path in the Host App and will process and reveal InfoObjs in proper 

15 context of the interface. 

Furthermore, based on the disclosed CHA architecture, a tool may be provided 
for searching through all the application's interface's text labels to help locate a GuiObj 
control hidden in the interface amidst possibly hundreds of controls. For example, if a user is 
interested in all controls related to Tab operations in a complex word processor, searching for 

20 the string "tab" displays or leads to all locations where the Tab parameters could be modified. 

The results of any of the interfaces mentioned above and subsequent Search 
21 15 is one or more returned Assistance Item Keys located in the database. Once the 
Assistance Item is located, a Guide Sequence's processing for a Sequence of a Fixed type 
occurs as follows. 

25 The set of one or more Assistance Item Keys is passed to the Sequence 

Manager Module 2025, shown in FIG. 12, which sets up a pointer to the set and which 
initializes its pointers to the first input Guide Sequence Key structure. Since the search result 
may include, for example, N number of items depending on the search criteria, the Sequence 
Manager function counts and processes each item until the list is completed, in step 2120 

30 upon which process flow returns to the Search Tool in step 2110. However if not complete, 
then a test is made for the presence of a Dynamic Guide Sequence in step 2122. If a Dynamic 
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Guide Sequence is being processed, then its initialization takes place in step 2123, and 
continues on to Guide Sequence initialization in step 2130 into the procedures and steps 
common to Fixed and Dynamic Guide processing. 

During the Dynamic Guide initialization, preparation is performed for later 
5 ChaSys Script Generation that produces an Opcode Stream, which is fed into the Main 
Sequence processor step. Here the output of the Search Tools i.e. GuiObj ID/Keyval is 
prepared for use in Dynamic functions that query ChaDB, in particular the Host App Model. 
This stage also serves as an entry point for the CHA Monitor Mode as described in FIGS. 15 
and 15A, as well as for the CHA Explore mode. If a Fixed Guide Sequence is detected, then 

10 its Script record is retrieved in step 2125, and upon delivery of the Script record from the 
ChaDB, the fixed Guide Sequence Record, which gathers code and pointers to sequence 
related data, is stored into memory. 

The Guide Sequence Initialization stage 21 30 sets the state of all major 
structures by setting sequence processing state flags, by reading global options, and by setting 

15 global state flags, and step 2130 also sets the Destination Structure used for verification of 
Guide Sequence completion. The pointers to the current Host App Model for the Primary 
Host Application are also set. This stage may also include a cleanup process during which all 
pending operations from possible previous sequence execution, especially any pending sub- 
processes, are cleared and any operation flags are reset, and principal execution pointers are 

20 also set. For example, used for asynchronous processes between different subsystems, 
ChaSysQueue is cleared of any pending InfoObj data which is sent to InfoObj Services for 
processing, or alternatively such data may be cleared. If a previous Guide Sequence is still in 
the Queue, this previous Guide Sequence is cleared, or alternatively a prompt is sent to the 
user for user intervention to process, resume, or abort the sequence. 

25 At step 2131a test is made for Dynamic Sequence Generation and if yes, the 

Target GuiObj is passed onto Dynamic specific functions 2132. Refer to FIG. 14A and its 
accompanying description below for further details. 

Continuing, it is determined if the CHA System is in Development mode. If 
so, the DevFlags Group is set. The DevFlags Group establishes the Development state for the 

30 CHA Run Time Modules and is a necessary component for the creation of new instances of 
CHA processes i.e. Sequences, Procedures, etc. with integration of all media data types for 
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expression and InfoObjs, on a synchronized schedule coupled with user input event handling. 
ChaSys Processor module pointers and structures are initialized with the initial Sequence 
Step Structure (SeqStepStruct) which is a data structure containing all data for the Nth Guide 
Sequence Step. Additionally a call to routines in the Host App Subclassing Object 
5 enumerates any of the Host Application's currently open windows or interface controls, and 
fills in system details to SysGuiObj structures. Such details may include operating system 
internal structures such as window handles, window class names, and other information. The 
handles to the Host App's main window and its principal menu items or iconic toolbars of the 
OS's physical GuiObj exist and so act as launch mechanisms for later Guide Sequence 

10 processing. The ChaSys Processor is now ready to begin processing the first Sequence 
SeqStepStruct pending the preparation and arrival of the SeqStep required resources. 

Once a record is completely loaded, or at least up to and including one 
completed Guide Sequence Step, the data expansion process begins 2135. Guide Sequence 
Steps allow for asynchronous operations in which a complete, logical portion of the data is 

15 loaded and is ready for processing. Subsequently, while one Step is being processed by the 
CHA engines and run-time components, other Steps can be requested and subsequently 
downloaded via the ChaDB Retrieval Module and I/O services. Capable of concurrent 
processing of numerous sub processes, the CHA System makes the downloading of multiple 
data sets of various sizes, transparent to the user in networked environments where data 

20 transfers rates are inhibiting. During the Expansion method in step 2135 in the Data 

Expansion Module 2035, different parts of the record are located, including the header which 
describes the type of data included in the record, and implicates pointers to its methods for 
expansion processing. 

Decompression can take place immediately, or optionally can store the data in 

25 compressed form in memory for later decompression on-demand. Other files may also 
remain in storage or may be made ready for transmission from a remote location, depending 
on CHA System configuration options. All record data is transmitted and transformed to the 
appropriate memory blocks and corresponding pointers are established. Other procedures 
executed are pointer construction and the building of indexes to facilitate searches or the 

30 location of records. 
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One of the memory blocks includes the Guide Sequence and Sequence 
Resource data. Sequence Resource data is either locatable via pointers, or embedded data can 
be included inside the script structures. Such pointers can also include references to existing 
Behavior Models found in ChaDB's contents, which greatly reduces the memory size of 

5 Guide Sequence Records. The Guide Sequence Record includes a mixture of several types, 
for example, InfoObj Data, User Custom Data, and Commentary or Expression Data, and 
may also include code to facilitate additional processing or to facilitate the assembly at 
different stages including responses to real-time analysis as a function of user actions or other 
calculations discovered only during user interactions. As the Guide Sequence script is 

10 processed via the Guide Seq Parser, tokens are extracted that map to language keyword 
operators. 

The following is an example Guide Sequence Record for a fixed sequence: 
Guide Sequence Record 
Global Options 
Header 



15 



// for global processing 

// contains record ID indexes, inter 



record data 



commentary 



Commentary Process Modifiers // modifies run time processing for 



20 



25 



stochastic 



opcodes 



determined 



Weights 



Generator Code 



SeqStepl Struct 
defined by a User 



30 



// generation 

// statistical influences on generated or 
// operations 

// optional, provides ability to generate 
// on the fly given a set of parameters 
// at run time 

// a Guide Sequence Step, usually 
// action 
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Script Code 



into opcodes 



the Host 



Host App 



10 supported 



recombined 



15 



GuiObj ID 



InfoObj 



Commentary Data 



SeqStep2 Struct 
Script Code 
InfoObj 

InfoObj Graphic Animation 



Expression 



20 



animation 



formats for 



25 



30 



Commentary Data 
User Custom Data 



SeqStep3 Struct 
Script Code 
GuiObj id 
InfoObj 

Commentary Data 
User Custom Data 



// opcodes or a script which is compiled 
// that determines interactive behavior on 
// Application 

// reference identifier to a GuiObj in the 
// interface 

// essential information in any format 

// by the expression engine 

// non essential commentary that can be 

// at run time 

// 2 nd sequence step 



// specific format supported by the 
// Engine that supports rich, interactive 

// User entered data in a variety of 

// individual customization 
// 3 rd sequence step 
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SeqStep4 Struct // 4 th sequence step 

Script Code 
GuiObj ID 
InfoObj 

5 InfoObj Graphic 

Commentary Data 

SeqStepN Struct // Nth sequence step 

10 SeqTerminator Code // ends the sequence 

The Sequence Script is subdivided into N partitions called SeqSteps which 
include a SeqStep structure usually, but not necessarily, corresponding to a Host App GuiObj 
which has an associated InfoObj ready for presentation with which the user interacts. Among 

15 other data, the SeqStep includes Host App state information that captures the state of the 
application at the time of creation of the Guide Sequence. For example, if a persistent 
GuiObj e.g. a checkbox was disabled during the time of the Guide Sequence creation, and the 
corresponding state information is stored into the Guide Sequence data structure S, and then, 
during execution of the same Guide Sequence the same checkbox is found to be enabled, then 

20 by checking the stored data structure S, the checkbox is enabled via mechanisms shared by 
CHA ICAP which simulates user interactions with the interface with a pointing device, to 
reproduce the target Sequence events and Host App's program states. 

From the Sequence Script's current SeqStepStruct, the active structures are 
located and evaluated as either a GuiObj or an InfoObj. If the active structures involve a 

25 GuiObj, the GuiObj is set as CurrentGuiObj. In addition, a SeqStepStructs may include any 
number and any combination of GuiObj or InfoObj references. The subsequent loading and 
execution of Sequence SeqStepStructs are controlled either by a user action or by a 
scheduling mechanism which can be parameterized and which can use heuristics to construct 
choices in real-time. 

30 The parser detects numerous Guide Sequence Script operators typical in a script language 
such as conditional processing using IF, THEN, ELSE statements. 

42 



WO 00/68769 



PCT/US00/12106 



In addition, being a group of operators with a corresponding set of parameters, 
the Host App Model Interface Function set located in the Host App Model Processor module 
is used to provide programming interfaces for external functions that manipulate or read the 
Host App Model 2140. By using these interfaces, many operations can be executed using 

5 small input data especially to locate logical or virtual Graphic User Interface Objects 
(GuiObjs) and their relative position within the application's interface hierarchy. These 
GuiObjs and their properties are used as keys to facilitate operations from simple position 
lookup to inference and the construction of automated procedures such as ICAP. 

One function of this programming interface group is the ability to enable the 

10 description of complex pathways in a very compact form and various operations on the Host 
Application Model. One such operation is handled by the function cFindGuiObj that 
receives, as an input parameter, a destination GuiObj Address of the Host App Model and 
then returns a pointer to a data structure record, which in turn points to details such as its 
ordinal position in the application hierarchy, points to pointers to its parent or other children 

15 GuiObjs, points to various attached InfoObj structures, points to pointers to User Custom 
Data, to Cyclic Redundancy Check (CRC) values for data verification, etc. In addition, 
another function cFindPath receives an input GuiObj destination and returns implicit details 
that can lead to the pathway to reach this destination as well as to retrieve various data 
structures from the Host App Model, such as InfoObj structures, along the same pathway. 

20 This function is also used in Dynamic Sequence processing. By using these functions that 
navigate the Host App Model, InfoObj structures attached to GuiObj records are also 
retrieved and with cStoreExecBuff, instances are created in the SeqExecBuffer for upcoming 
processing commencing beyond 2150. 

The CHA system includes a mechanism to assemble optional Commentary 

25 objects in which the commentary represents a humanizing and optionally non-essential 
dialogue that is presented to the user either concurrently or sequentially as the user proceeds 
through Guide Sequence SeqSteps and interacts with Host Application GuiObjs. For 
example, if, during processing of the Guide Sequence, the directions as presented either 
through a character or text, or voice, etc. indicates for the user to press a GuiObj, after the 

30 user responds with a mouse click, keypress, or other actuating input or selection, the CHA 
System may respond with a simple exclamatory "Good!". The benefit of this mechanism is 
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that such commentary provides feedback to the user and contributes to the learning 
experience. The CHA System includes such a mechanism not only for providing such 
commentary, but also to create phrases and sentences dynamically to prevent the tedious 
repetition of phrases. 

5 At step 2145, an assembly of commentary objects is performed by the 

Commentary Assembly Module 2045 that receives an input structure framework from the 
Host App Model Processor. The Commentary data set may include different target categories 
of fragments that are grammatically compatible and interchangeable according to a set of 
specific language rules. The Commentary Set may also include categories that are organized 

10 phrases that can be assembled in run-time into many different combinations to generate an 
extensive variety of phrases or expressions for the Guide Character, which is any animated 
character or other graphical representation interleaved with the presentation of InfoObjs. 
Commentary sentences and phrases can be created and rendered according to the selected 
data format, for example, by concatenating ASCII text or assembling sound fragments as in a 

15 play list or files within the Expression Engine. 

The CHA system and method uses, for example, several Commentary 
Categories: Non-Directed, Directed, and Transitional. The Non-Directed category includes 
comments which are not in reference to any particular interaction with a GuiObj or action by 
the user. On the other hand, the Directed category includes comments associated with a 

20 reference such as the user clicking on a specific GuiObj control. The Transitional category 
serves as a connective function presented by the Guide Character e.g. an animated character 
or group of characters. These commands can be performed at the start or finish of a Guide 
Sequence or may provide transition and natural cadences between Sequences or pauses 
during the Sequence. 

25 Furthermore, the Commentary Categories may, in turn, be subdivided into 

different intentions, for example, to convey various moods, intonations, or expressions that 
generally attempt to assemble or build variety and character into the presentation. 
Assembling these phrases at run-time has the distinct advantage of reducing data production, 
and does not require the preparation of every possible phrase, and such assembling of phrases 

30 also contributes to the "personality" of the learning or interactive experience. 
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The following Commentary Categories may be Directed commentaries to 
specific GuiObj references or responses to action: Positive Humorous, Negative Humorous, 
Positive Encouraging, Negative But Encouraging, Positive Neutral, Negative Neutral, 
Positive Exclamatory, Negative Exclamatory, Positive Learned, Negative Learned, 

5 Next Step, Previous Step, Incremental Iterative Responses, and Different Moods from 
lethargic to energetic. 

The Transitional category includes commentary associated with a State 
Transformation, with such Commands including: Stop Now, Start Now, Finished, About to 
Begin, Look For..., and Reminders or Tips. 

10 The Non-Directed category have no particular reference, and may include 

commentary such as: Sarcastic, Verbose, and Humorous. 

Each commentary may be interchangeable from a set of word or phrase 
choices to another within a category and sub-category. 

Each Category is further divided into at least two parts: a Determinant Word 

15 Operator and a Phrase or Word Variable. The Commentary Assembly module 2045 locates a 
Determinant Word Operator (DWOp) and extracts an identification (ID) value which is 
passed to the Commentary Assembly module 2045 where a Phrase or Word Variable is filled 
in. A weighting system may be optionally implemented which affects the probability of word 
selection and so further adds an additional dimension to influencing phrase construction. 

20 Once Commentary data is prepared and stored in the Commentary output data buffer, the 
Commentary data stored in the SeqExecBuffer using a call to the function cStoreExecBuff 
located in the Procedure Execution Module 2055. The function cStoreExecBuff is called to 
create instances of the Commentary data in the SeqExecBuffer. 

At this point in the Guide Sequence execution process, essential data elements 

25 have now been prepared, flags are set, and processing now begins in the Main Sequence 
Processor, using the component ChaSys. First a test is performed in step 21 50 to see if the 
current Sequence is complete. If the current Sequence is determined to be complete, then the 
process proceeds to another test to determine if there are additional Sequences to handle. 
Depending on the search tools used in step 21 10, some searches can result in multiple 

30 Sequences such as by searching for all Tab commands located in a plurality of locations in a 
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word processor. If the current Sequence is incomplete, processing is directed to ChaSys 
which is part of the Main Sequence Module 2050. 

ChaSys uses different synch objects, as well as known operating system 
mechanisms to coordinate the passage of data across different OS processes or threads. The 
5 ChaSysQueue is used for managing asynchronous processes between different subsystems. 
When a request is made to a subsystem to process data, instances are created in the 
ChaSysQueue and packets of data are exchanged with different subsystems using, for 
example, the Hooking Function located in the Subclassing Object 2060, and also using the 
Expression Engine located in the InfoObj Services Module 2070. 

10 Since processing in the Host App Model Processor 2040 retrieves from the 

Host App Model the designated Interface Pathways through the Host App's interface, 
SysGuiObj Initiation occurs for the mapping process between the referenced GuiObjs from 
the Host App Model to the SysGuiObj data provided by the operating system. A native 
operating system event occurs when any of the Host App's Windows are opened either by 

15 some CHA System automated process or by the user, and the Hooking Function 1415 detects 
the OS's window creation message e.g. WMCREATE in "MICROSOFT WINDOWS". 
Additionally for existing windows, the CHA System contains an enumeration function that 
identifies all windows and controls which it can locate. In both cases i.e. for the creation of 
or for existing objs, attributes of the GuiObjs are extracted e.g. window handles, class names, 

20 etc., and stored in a transient storage structure (SysGuiObj) bound to a GuiObj. Double 

linked pointers are also set between the SysGuiObj and its corresponding, mirrored GuiObj in 
the Host App Model. By using these extracted GuiObj attributes, lookup keys, such as the 
GuiObj Key, are built and used to locate a corresponding record in the Host App Model. 

Control of the Host Application's interface occurs using any CHA Mode 

25 Process such as the Guide Sequence Process or ICAP, and involves the automatic location 
and control of any Host Application GuiObj in a series of actions such as the actuation of a 
menu item, followed by the opening of a dialog box, the actuation of a button inside the same 
dialog box, etc. Such steps are accompanied by the presentation of InfoObj s synchronized 
with user GuiObj interactions. An internal operating system or procedure enabling data 

30 structures, such as window handles in "MICROSOFT WINDOWS", are made accessible for 
use by the Operating System and are needed to gain control of a given GuiObj. The choice of 
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one or more GuiObjs is determined by scripts or real-time navigation through the Host App 
Model using symbolic interfaces or implied pathways resulting from deductions, heuristics, 
or calculations. 

This enabling OS structure, which can be any of a number of types, depending 
5 on the OS and the application, allows control of the corresponding GuiObj along with access 
to certain object properties through the OS such as a window control's position, size, window 
style, class name, etc. The order and specific GuiObj, e.g. a button or icon, are all dictated or 
implied by the specific CHA data files from the particular Mode of Operation coupled with 
retrieval of Interface Pathways through the application's interface hierarchy in the 

10 fundamental Host App Model. To emulate the series of user-spawned application events, and 
to maintain guidance of the user through the Host App interface, it is therefore necessary to 
invoke, in a series, each targeted GuiObj via the operating system's mechanisms. 

In this case, the CHA System opens each window or window control via part 
of the CHA ICAP Process when a windows message is generated and sent to the Target 

15 GuiObj via an Application Program Interface (API) call to a function similar to the 
"MICROSOFT WINDOWS" SendMessage function using the Target GuiObj's window 
handle. Once accessed and visible, the required operating system entities are then extracted 
and stored into SysGuiObj structures. Pointers are also set to their counterpart, "mirrored" 
GuiObjs within the Host App Model framework. With SysGuiObjs in place, bi-directional 

20 communication occur between CHA Host App Model processes located in the Host App 
Model Processor 2040 and respective GuiObjs in the logical Host App Model's framework 
and objects in the live, Host Application's graphic interface. 

As shown in FIG. 13, processes performed in steps 2155 and 2165 interact 
21 60 with the SeqExecBuffer. A parser routine identifies all data formats and tokens 

25 included in the buffer, and routes each identifiable entity to the proper input and eventually 
service routines. The triggering and response to various events, both internal and external to 
CHA, interact with time-based data types such as sound files, animations, and the 
presentation of information. For example, a pictorial character is displayed on a screen and 
messages, either audio and/or visual, are output to the user describing a GuiObj, and the 

30 character may be modified to be pointing to the GuiObj with a depiction of a hand of the 
character. As the hand raises or otherwise moves to a pointing orientation and/or position, a 
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highlighting process is performed to highlight the GuiObj. Such audio and/or visual events 
are properly synchronized to provide a smooth animation or motion in conjunction with the 
messages to the user. 

At this point the SeqExecBuffer has been filled with necessary components, 
5 pointers, and operators to allow the execution of the series of SeqSteps to perform the 
completion of the Guide Sequence Process. With the work of loading, expanding, and 
assembling data selection components having occurred, a next task includes coordinating the 
Guide Sequence's automatic processes with the user's actions being translated to OS- 
generated events that require monitoring and coordination with the CHA System internals. 
10 The ChaSys component, being included in the Main Sequence Processor 

Module, manages the principal coordination in step 2160 of all subclassing processes, 
SeqExecBuffer contents, and subprocesses occurring at this stage. Furthermore, the Behavior 
Model, as defined by ChaSys Scripts, determines the style of interaction and the presentation 
of information. 

15 For performing the processes involved in the interaction with the contents of 

the SeqExecBuffer, the contents of the SeqStep sets the boundaries governing all logic of 
internal and external occurrences. Either before or during each SeqStep process, the Main 
Sequence Processor marshals the input data set by filling input buffers with InfoObj and 
Commentary data, by processing relevant SysGuiObj structures, by filling such SysGuiObjs 

20 with extracted data from the operating system, by integrating User Custom Data, etc. Each 
SeqStep is executed and complete its logic until processing proceeds to the next SeqStep. 
Additionally, optional mechanisms are used to provide forwards or backwards navigation 
through SeqSteps as well as random access, within certain behavioral restrictions of the 
operating system. 

25 FIG. 14 illustrates the steps 2155, 2160, 2165 and 2170 of FIG. 13 in greater 

detail, where the data collected in the SeqExecBuffer 2203 is the input in an interactive stage 
in which user actions are intercepted. The ChaSys Component 2204 performs coordination 
and management of processes, for example, to trigger, suspend, or control different processes 
during the operation of the steps shown in FIG. 14. 

30 CHA Events 22 1 0 are generated by the Hook Function 2206 and by ChaMap 

processing 2209, and such CHA Events are passed to the ChaSys Script Processor 2212 
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which maintains multiple processes, including a scan loop routine to detect interrupts or 
messages. Once such interrupts or messages are detected and received, such interrupts or 
messages are evaluated and routed accordingly. For example, a message may include the 
request by the user to abort the CHA System, may include an indication that the user has 
5 responded to an InfoObj and is interacting with a Host App interface object, or may include 
an indication that the user has minimized the Host Application Window, and so the 
processing and different CHA application threads are to be suspended in order to not use 
CPU processing time. 

Concurrently the ChaSys Component executes associated Parser routines 2215 

10 to evaluate each token in the Sequence Scripts. When such tokens are mapped to Script 
keywords, the corresponding script-token-specific functions are executed. As script code is 
scanned and processed, inevitable references for resources occur, in which resources can be 
any of the types and formats previously described. Such requests are handled by a Resource 
Locator section 221 8 of ChaSys which operates to successfully fulfill all requests for the data 

15 types needed as input for the Expression Engine. 

Once such data types have been located, the Dispatcher reads the data structure 
headers, determines the data type, and calls the appropriate service routines to render the 
demanded form of expression. This service may be performed as a registration of the request 
due to the need to transfer data from a remote location. In some situations, this may prevent 

20 the completion of the currently executed record entity while the request is being completed. 
In other situations ChaSys may allow certain adaptive steps to take place even without the 
pending data. While most user actuations may be correct, ChaSys uses its Sequence 
Exclusion Processing components 2224 to parse CHA Events and to perform comparisons to 
rules established in the CHA Sequence Script or the Behavior Model being referenced in 

25 order to determine incorrect Host App GuiObj responses, or inappropriate pathways. This 
excluding action handling component makes decisions based on the configuration of the Host 
App and its current window state. 

FIG. 14 also shows the relations between Data items 2227 through 2239, and 
their respective Services 2242 through 2254. Referring to such data items and services, in the 

30 Guide Sequence method, multitasking Services are initialized and prepared for input, with 
such multitasking services including the various functions that handle each of the different 
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data types including: Commentary 2227, InfoObj 2230, UserData 2233, Graphic Data 2236, 
animated Guide Character Data 2239, and their respective Services 2242, 2245, 2248, 2251, 
and 2254. As the selected Expression Engine 2257 maintains and schedules many concurrent 
tasks involving the IVM multimedia engine 2263, the Generic 2260, or the 3D 
5 Graphics/Sound Engines 2266, each service is allocated a share of CPU and hardware 
subsystem processing resources, such as the video and sound processors that manifest each 
data type in synchronized, scheduled events in conjunction with the computer's input. Since 
the Graphic and Character Data are visual and therefore utilize appropriate data types native 
to the operating system such as Graphic Device Interface (GDI), Application Programming 

10 Interface calls, or bitmap (.bmp) files in "MICROSOFT WINDOWS", the graphical input 
data is packaged in file formats indigenous to the Expression Engine's drawing techniques 
such as the IVM specific animation files. 

A call is made to the Graphic Service 2251 which handles the Highlighting 
Function as well as other effects. To accomplish this task, the location of the Target GuiObj 

15 is found from the structure in the SeqExecBuffer. Next a call is made to initialize a 

concurrent task for the character 2254 where its current position is displayed relative to the 
Target GuiObj which is referenced by the InfoObj. Also, in the SeqExecBuffer, the current 
SeqStepStruct is examined for any InfoObjs. If an InfoObj is detected, its pointers are sent to 
the InfoObj Service 2245. During the processing of the InfoObj, the concurrent character or 

20 its equivalent refers to the GuiObj in a synchronized fashion such that the character points to 
the referred GuiObj at the appropriate moment. 

A check is also performed to determine the presence of any User Data 
previously attached to the Host App Model structures or elsewhere or to the current InfoObj 
and is forwarded to its service function 2248, such User Data is present. The user is 

25 prompted to respond to directions or information given in the InfoObj using the computer's 
input devices for example, a keyboard, a mouse, a voice response system, etc. Actuation of 
any of these devices generates "MICROSOFT WINDOWS" messages that are intercepted by 
the CHA Hooking Function 2206, and then passed to ChaMap 2209, which intelligently 
filters a multitude of operating system generated messages. The ChaMap 2209 interprets 

30 such messages as CHA Events, and then dispatches a reduced set generated from such 

messages to the Main Sequence Processor including the ChaSys component. Upon receiving 
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this Hooking Function and ChaMap-generated CHA Event set 2210. ChaSys evaluates the 
CHA Event set, and if a user's action is detected to be outside of the Guide Sequence's path, 
the Sequence Exclusion Processing section 2224 further evaluates and generates directives 
leading to an appropriate negative response and feedback. 
5 From each of the Services of the different data types, the final manifestation 

stage is performed which renders each data type for audio, sound and both static or animated 
graphics. In this phase the Expression Engine 2170 uses the selected Multimedia 
multitasking engine such as the IVM Object, and for each of the input data types the 
Expression Engine 21 70 uses selectively the different hardware components to display or 

1 0 play sound, such as text-to-speech engines ot other audio mechanisms, as well as streaming 
network audio, for output to the user 

In addition, ChaSys can trigger a Guide Sequence Reset command for the 
reprocessing and reassembling of all data and code components in the SeqExecBuffer to 
accommodate situations requiring data updating, such as changes in strategy, or path 

15 directions if the user wishes to branch away from the Guide Sequence or to invoke the 

Explore Mode. Once the new pathways are determined through a variety of mechanisms such 
as by use of the Interface Map Tool, reconstruction including the new elements occurs in 
stages 21 05 through 21 70. Another cause for a Reset command to be executed may occur if 
the CHA System detects changes in the Host App interface, and so the operating system 

20 message is detected which triggers an updating of internal GuiObj structures, and hence the 
SeqExecBuffer contents are also updated. This exemplifies the adaptive response and 
dynamic behavior of the CHA System versus the inherent fixed architectures and a hard 
coded scripting systems and methods approach. 

The expression component i.e. interactive and multimedia processes can be 

25 implemented in many ways, for example, using script languages, C/C++, or any other 

computer language that controls visual or aural processes. In an illustrative embodiment, the 
CHA system and method uses a scripting language which is a "virtual machine" that performs 
very efficiently in a wide variety of environments and is portable to different operating 
systems. Other engines for expression can also be used. The use of script languages is a 

30 common technique for simplifying a set of known operations in a problem domain and can 
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automate many operations normally requiring specialized knowledge or considerable manual 
programming effort. 

In specific Dynamic CHA modes of operation, direct scripting is not needed. 
However, optional scripting for specific tasks is still made available for flexible development. 

5 In the CHA System there are two accessible layers that control the Sequence execution: in the 
ChaSys component and in the Expression Engine using the IVM Class object. A multi-tiered 
architecture using, for example, scripting, provides for a separation of functionality and eases 
production, and alternatively allows portability to various operating systems and operating 
environments or different application frameworks. The CHA System's run-time components 

10 utilizes the Sequence ChaSys Script partitioned into a series of SeqStep Scripts as well as the 
Expression Engine's IVM Scripts. 

The benefits of this multi-level feature includes the facilitation of a more 
simplified, modular design, as well as ease of maintenance, the development tools interface, 
and the more convenient management of all InfoObjs and CommentaryObjs and other types 

15 of data. An additional benefit of use of the Script language is its flexible delivery and 
dynamic, modular, and compact properties. For example the Sequence Scripts can be 
independently compiled prior to use, or compiled dynamically, just-in-time or on-demand, 
and furthermore, can enable the rapid switching of different sets of data such as InfoObjs in 
requirements for applications such as in different language presentations. In alternative 

20 embodiments, scripts may optionally be replaced by any of a number of different computer 
languages as known in the art. 

Note that while the description focuses on the creation of Guide Sequences on 
single applications, the invention also can coordinate events in multiple Host Applications 
with a single ChaSys Multi Application Script. This is accomplished by creating multiple 

25 instances of Hooking Functions, one for each application and routing the hooking messages 
which map to analyzed CHA Events. Such analyzed CHA events are then passed across 
applications to a single collective functioning entity named the Multi ChaMap Processor that 
resides within a single "Super" CHA run-time instance. This Multi ChaMap Processor 
handles messages across multiple Host Applications, and passes them on to the Super 

30 CHASys Engine instance which interprets actions for the different applications by further 
expansion of GuiObj ID's which contain a prefix application identifier i.e. codes referring to 
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CHA registered applications. The GuiObj Key values are furthermore mapped to the 
appropriate Host App Models for their respective Host Applications. The handling of 
multiple application contexts is a natural extension of the inherent architecture of the singular 
CHA process example given and may be implemented in a manner known in the art. 
5 As described herein, the disclosed CHA system may operate using Fixed 

Guide Sequences. The CHA system may also operating using Dynamic Guide Sequences, as 
described herein, which may be considered dynamic counterparts to the Fixed Guide 
Sequences. 

While sharing many of the same modules and routines as in Fixed Guide 

10 Sequences, Dynamic types differ in that the Dynamic Guide Sequence rapidly responds to a 
user's spontaneous navigational commands while interacting in a live Host App Session. 
Unlike prior art assistance systems, the disclosed CHA System and method in its Dynamic 
Mode of operation does not require the preparation of every path permutation to every Target 
GuiObj, and do not use fixed content Sequence Records retrieved from a set of fixed data 

15 storage files. Instead, single Target GuiObj s on the Interface Map Tool can calculate and 
generate P Paths with N SeqSteps, where P = all possible permutations within the rules of the 
given Behavior Model, from N=0 to MaxSteps, using path-finding heuristics and algorithms. 

The value for P may be extremely large such that a user may use the CHA 
application for a very long period of use and never exactly repeat all the Commentary or 

20 InfoObj combinations of events presented in the Dynamic Guide Sequence. The CHA 

System and method also create Connective Language Phrases from the Commentary Data Set 
and its mechanisms to set up the execution of InfoObj s, in which Connective Language 
Phrases are spoken or textually rendered and function to finish an action or to introduce the 
next user action. Additionally, multiple Target GuiObjs are supported for multi-path 

25 generation and the compilation of a series of Sequences. 

FIG. 14A describes the related processes of the Dynamic Sequence. First, a 
Symbolic Map Interface (SMI) is rendered and displayed to the user in a CHA Search Tool 
21 10, shown in FIG. 13, that provides visualization of the interface. In the SMI the user 
inputs a selection i.e. one or more Target GuiObjs, with the input device, such as a pointing 

30 device or keyboard, and returns the corresponding GuiObj ID 1905 which is an identifier for 
a GuiObj that is referenced in the script code or by an opcode set. The GuiObj ID is also then 
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mapped to a GuiObj Key, which is used for Host App Model lookups. The CHA system and 
method proceeds to execute the function cFindGuiObj 1910, such that the Host App Model is 
searched to locate the Target GuiObj record in the interface model description. A test is made 
191 5 to verify that the destination GuiObj is found. If not, processing in step 1920 is 
5 performed to resubmit, redo or perform another search, and then to return to the Search Tool 
in step 1965. If yes in step 1915, then a second call to cFindGuiObj finds the record of the 
Current Location of the user, or the user can instead specify a specific starting point anywhere 
in the Host App. 

Once the starting GuiObj is found, the cFindPath(A,B) routine is performed 

10 1925, in which A and B are start and end locations, respectively, so that the cFindPath(A,B) 
routine is used to determine the best path, or alternatively or a set of alternate paths i.e. a 
Dynamic Paths Set 1930, between the Current Location and the Target GuiObj. The 
cFindPath(A,B) returns a set of cPathVectors which is a structure that contains all relevant 
pointer information to Host App Model records. 

15 Initialization of the Dynamic Guide Sequence Module occurs, which also 

includes the main loop processing and the sequence termination mechanism. Using the 
cPathVectors Structure found by cFindPath, a Dynamic Guide Sequence Record (DGSR) is 
created 1935, including a series of Transport GuiObjs, in the SeqExecBuffer. In addition, 
data structures are created with initial SysGuiObj information and their links to GuiObjs, 

20 including data from the Host App Model and including GuiObj Link information. 

Additionally ChaSys Script source code or binary opcodes are generated and inserted into the 
ChaSys Processing Buffer 1940 for processing during the Sequence execution. The Dynamic 
Guide Sequence Script or its associative opcode set is automatically generated and inserted 
into the buffer for later processing by the ChaSys Script Processor. 

25 Continuing, in 1945 InfoObjs are retrieved according to the InfoObj Detail 

Level parameters which determine what data sets are selected at each SeqStep during the 
current Dynamic Guide Sequence (DGS) Process. Pointers to the Host App InfoObj Set are 
determined for Sequence Processing and stored into the SeqExecBuffer. Optionally instead 
of pointers, actual data can be packed into the SeqExecBuffer for collection, compression, 

30 and export such as in the case of server/client delivery. Similarly at 1950 the Commentary 
Levels are used in preparation of the Commentary object data retrieved from ChaDB. 
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Interleaved into the SeqExecBuffer DGS record and the SeqSteps, the Commentary data is 
ready for processing and depending on options, either as pointers to data or the actual data 
which can be packed and prepared for network transmission. Next, the SeqExecBuffer is 
initialized 1955 and prepared in a manner compatible with components that are shared with 

5 Fixed Guide Sequence Processing as described previously. Next processing returns 1965 to 
FIG. 13 and proceeds to 2150 for execution of the generated Guide Sequence. 

The CHA System's Explore Mode also utilizes these Dynamic mechanisms. 
In the Explore Mode, instead of laying out the entire DGS Record containing N SeqSteps 
ahead of time, requests are made to the Symbolic Map Interface on an incremental, step-by- 

10 step basis. Each step, as a SeqStep, is completely executed one at a time, and when each step 
is completed, the CHA system and method return to fetch and respond to the next directive 
from the user for a GuiObj selection. The user can explore parts of an application or the 
entire application, and data is presented at different requested levels while the user interacts 
with the Live Application. Certain restrictions may be implemented which prevent 

15 inconsistent operations during this mode to preserve the integrity and synchronization of the 
application state with the InfoObj Set. During the exploration, choices can also be made as to 
whether Host App parameters are actually set or not. 

The CHA System may also optionally undo any changes made, may enable 
only Transport GuiObjs, or may, for example, respond to the action of a user to an accidental 

20 actuation of the Cancel instead of the OK button in a dialog box. Variations in the use of the 
application may be viewed in the Explore mode, such as options for the most direct route, 
different levels of detail, verbosity, the altering of connection phrases from the Commentary 
Set, etc., and such variations may be adjusted with parameters through a CHA controls 
interface. When the DGS record is created and filled, program flow proceeds to step 2123 in 

25 FIG. 1 3 where it begins a Dynamic type processing. 

In Explore Mode, path finding algorithms are applied to the Host App Model 
to find a path in real-time. An input flag is set which determines the InfoObj Detail Level 
during exploration, or which determines a traversal operation through the path. The InfoObj 
Detail Level (IODL) determines which level and parts of the database of InfoObjs are 

30 accessed, and which data is retrieved. The IODL determines the scope of its description. For 
example, if the level is a Summary value, then the InfoObj retrieved include general 
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information about the GuiObj e.g. information for use in a message to a user such as "This 
dialog box contains a set of controls for determining the format of a word processing 
document." As the path is traversed and data is retrieved from the ChaDB, InfoObj data and 
other elements such as Non-Essential, Negative-Response, or Positive-Response Commentary 
5 can also be constructed in real-time, depending on global and parameter settings, and 
presented to the user using the IVM class engine for expression. 

Finally, while a given Guide Sequence presents itself as a fixed series along a 
path of GuiObj s, with a CHA control, this fixed path can be broken and departed from via 
auxiliary alternative methods. The CHA System first preserves its current state with a stack- 

10 like Push State function, and then unlocks entry to other parts of the application not on the 
current sequence's GuiObj Path, with run time mechanisms for the generation of GuiObj 
Paths along with parameters controlling InfoObj Detail Level retrieval. Such operations may 
be performed either through the entry to Explore Mode, or through Dynamic Guide Sequence 
system processing. With a CHA control, the user can choose to return to the initial Guide 

15 Sequence path after a Pop State command is executed and resume subsequent SeqSteps till 
the series is completed. 

Another major CHA mode, the Monitor Mode's purpose is to identify logical 
patterns from a set of user input actions, and to execute one of a number of functions, 
including: automatically create CHA Guide Sequences, construct ICAPs, or generate 

20 Inference Actions. When the CHA Monitor is activated, the CHA Hooking Function, which 
subclasses windows intercepts and monitors all user actions with the Host App. Every action 
or a subset of GuiObj actions are recorded. For example, for every mouse left button click on 
an interface button or any other GuiObj interaction, a CHA Monitor Event data record is 
stored into a separate monitoring database. A Session Monitor Data Set thus accumulates 

25 over a user set time period. Either at a scheduled time or invoked by the user, the CHA 
Monitor subsystem prompts the user, and upon the user responding affirmatively, causes the 
CHA Monitor process to begin a compilation and analysis of the accumulated Session 
Monitor Data. 

Processing of the Session Monitor Data is performed in cooperation with 
30 ChaDB the latter being built by the CHA developer with the CHA Development Environment 
(CDE). CHA Procedures that are a result of Session Monitor Data analysis and later 
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Sequence generation processing, on demand, assistance can be invoked, and intelligent 
options can be implied deduced and generated. The CHA System can provide the user a 
number of choices such as suggestions of faster ways to accomplish repeated tasks, or can 
offer ICAP commands i.e. automated procedures that are automatically assembled by the 

5 CHA System and included in the ICAP List for random ICAP Command access. These 
automatically built procedures can replace or supplement the familiar "Record Macro" tool 
available in many of today's applications. The standard Record Macro tool allows for the 
recording and reproducing of a user's actions for a sequence of steps. While still offering 
recording facilities, the CHA System's ability to generate accelerated procedures for 

10 specifically Target GuiObjs automatically is advantageous in its ability to perform such 
procedures dynamically in real-time. 

There are three stages during CHA Monitoring: Collection, real-time Context 
Evaluation, and Non-Real-Time Context Evaluation. The Collection stage stores the user's 
actions for later analysis. The real-time Context process consults the Host App Model in 

15 order to derive contextual suggestions for using the application, or embeds markers into the 
stream of actions being monitored. The Non-Real-Time Context process can perform more 
in-depth analysis and multi-pass processing on the collected data set gathered during the 
Collection Stage to produce reports, in context assistance, or automatically constructed 
Accelerated Procedures. 

20 Referring to FIG. 1 5, program flow begins with the launching of the CHA 

Monitor components. The Host App Model 2505 is stored in the ChaDB, and includes a Host 
Application Model i.e. an optimal interface structure for the CHA Monitor Mode, referred to 
here as the GuiFramework which is referenced for both navigation and intelligent inference. 
Besides representations of all instances of every GuiObj in the application, this framework 

25 also contains all link and connective information such that traversal actions, symbolic 

representations, and other operations can be processed or generated. Data structures 2530 are 
initialized and include a current Session Structure 2510 and a GuiFramework 2515, a Session 
flag to set the state of the GuiFramework, and a Session Identifier. The Session Structure is 
the global structure that organizes data events being collected during the Monitor Phase, 

30 including data events pertaining to Session Control Information. Additionally, the 
GuiFramework reflects the Host Application Interface, and provides a logical description 
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included in the Host App Model, which is referenced during the evaluation stage of user 
input. 

Upon completion of initialization, a first test is performed to verify that the 
Hooking Function is ready for operation. If so, processing proceeds; otherwise, the Hooking 

5 Function is injected into the Host Application 2540, as described in greater detail with 
reference to Hook programming in conjunction with FIG. 7. During the Collection Stage, 
user input is polled 2550, and if the user actuates any mouse button or keyboard key, except 
for mode control codes such as an Exit command 2555, an Action Structure is created 2560 
and processed by EvalAction 2565. The Action structure includes Status flags used to mark 

10 structures for tracking continuous paths during the analysis phase. The module EvalAction 
2565 performs rapid intermediate level evaluation that also facilitates later analysis. This 
evaluation may be performed minimally for optimal processing speeds, since such 
evaluations may cause interference with Host App performance. 

The EvalAction module also has a second real-time mode of operation that 

15 evaluates user inputs in real-time rather than in a post-operation, non-real-time processing. 
Capabilities of the real-time module depend on the computer environment, the type of 
application being monitored, and the requirements of the application. The Main Ref State 
may be a default, basic starting position in the Host Application retrieved from the 
GuiFramework from which all action series and potential sequences can be referenced. The 

20 Main Ref State acts as a reset point for CHA Sequences in an attempt to decipher start and 
end points for potential CHA Sequences. When the Main Ref State is detected, the 
EvalAction module sets the Main Ref State flag in the current Action structure. 

Another function performed in the EvalAction module is to search for 
potential Sequences that can be later processed as automated procedures. This is 

25 accomplished by searching for Continuities, which are a series of input steps that pass 
through consecutive portions of the interface hierarchy as described in the GuiFramework 
contained in the Host App Model, that lead to Parameter GuiObjs which are interface objects 
for setting the Host Application persistent state e.g. a checkbox. As an Action Structure is 
evaluated, it is flagged and a reference to it is placed into a Potential Sequence Tracking 

30 index which is used in later post-processing for ICAP verification. The Action event is then 
stored 2570. A test is also made to determine if the Memory Buffer A is full 2575, and so to 
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flush the buffer 2580 into a permanent file i.e. the Session Record. This process repeats until 
the end of the session. 

Referring to FIG. 15A, from the Host App Model, the GuiFramework is first 
extracted from the database and placed into available memory structures. The Host App 
5 Model's GuiFramework includes an application's interface controls, including the Transport 
GuiObjs that reflect the different navigational states of the Host App. The initial processing 
stage of the GuiFramework expands the data into a run-time form. The Host's 
GuiFramework 505 is normally maintained in a compressed form to reduce resource 
requirements, and, upon loading, the GuiFramework is then expanded via implicit rules into 

10 an interlinked, indexed network structure in memory which prepares it for fast random access. 
Furthermore, depending on the size of the GuiFramework structure, portions may be cached 
on disk or stored in a network-connected database, with a memory resident cache index 
mechanism used to manage the loading of data from other locations e.g. disk to memory or 
across networks as required on demand. 

15 As shown in FIG. 1 5 A, header information 5 1 0 is provided which includes an 

Initial Index which was created during the Action Collection recording process. The Initial 
Index points to a rough form of the data which identifies Sequences or potential Sequences 
detected during recording. Such identification reduces subsequent analysis since the data is 
already divided into individual sections with some degree of accuracy instead of having the 

20 data maintained in a monolithic block. Additionally the user can input optional directives 
during the input process which can assist later analysis. The next stage involves 
normalization 51 5 of the Session Monitor Data (SMD), which employs a number of 
processes. One process is the removal of extraneous pointer device movements, meaningless 
keystrokes, and/or Action structures, as well as "non-event" Actions such as when a dialog 

25 box is opened, but in which no Host App parameter is altered. 

Other actions are separated and removed, such as actions which involve Tool 
Bars or controls that execute some function with a single action, for example a mouse click or 
a keystroke accelerator. Any records of interactions with toolbar GuiObjs are also separated 
out of the SMD set. From detection of this type of Single Action Event 520 its structure is 

30 flagged and eliminated 525 from the captured Monitor data. From this SMD set, the 
Monitor's processing goal is to identify a Sequence which is a series of Actions that either 
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accesses or navigates a useful i.e. frequent GuiObj Path or alters one or more parameters in 
the Host Application via a Parameter GuiObj. The latter action usually leads to a Host App 
state Change. Therefore any series of actions that reaches a Parameter GuiObj, for example, 
in which a parameter is changed, is marked (SeqEndFlag = 1) as a potential Sequence. 
5 Further investigation of neighboring GuiObjs reveals the extent of the current 

potential Sequence. A probing process to find out if adjacent Actions go further into 
continuous levels of the Host App's interface structure i.e. extended 530, then the current 
Sequence Chain is expanded to include the new structure 535. This probing process 
continues until the Sequence Chain is determined to be complete. When done then a check is 

10 made for a Parameter GuiObj being modified. A few additional checks are made to verify 
that this is a true Sequence endpoint i.e. an investigation seeks a break of Continuity in the 
Action Structures. Note a break of Continuity is defined as a series of actions where a 
Parameter GuiObj is set or altered and followed by a series of Cancel or OK buttons 
(Transport GuiObjs) that causes a return to the Main Ref State of the Application. The 

15 GuiObj's Interface Level Position (ILP) is the value corresponding to any given GuiObj's 
relative position in the hierarchy of the Host App's interface as described in the 
GuiFramework. The GuiObj's are given an ILP Value 540 e.g. zero may be the highest level 
("root"), while N > 0 is the largest value and represents the deepest level of the application 
interface. 

20 The Monitor Processor first attempts to locate all Action Events with the 

highest ILP value i.e. GuiObjs located at the deepest levels of the Host interface. The 
intermediate pattern being sought involves any Action Structures with Continuous Properties, 
such that Continuous Properties are values within the Action Structures that represent 
proximity 545 i.e. consecutive ordinal positions within the Host App GuiFramework, for 

25 example, paths along the Transport GuiObjs. Conversely, a series of Actions Structures is 
considered to be discontinuous if it has "breaks 5 * in consecutive ILP values or if it includes a 
series of Cancel/OKGuiObj interactions. 

Inspection of the Transport GuiObjs traversed, along with any Parameter 
GuiObjs, reveals any continuous Actions 550. Other Action Structures with Continuous 

30 Properties might be a series of interactions with one or more controls contained in a group i.e. 
a dialog box. Those Action Structures with the highest ILP values 555, at the deepest 
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interface levels, are flagged and marked as possible Target GuiObjs. The surrounding 
proximal Action structures are also inspected for further logical continuities. Target GuiObjs 
and their associated Action Structures with Continuous Properties are indexed or collected, 
and are further processed to produce generated Sequences output. Once created, a check is 

5 made for a potential Sequence 560 and if identified it is sent to a list for later inspection or 
processing by the user. Generally, the Monitor Analysis tries to locate all Continuous 
Actions to identify their starting and ending points. Heuristics may be used to determine 
these start/end points based on their logical "degree of continuity" in respect to the 
GuiFramework. The previously described properties of Continuity, ILP Values, and the 

10 detection of Parameter GuiObjs are used in the detection process 560 of Sequence candidates 
where if found they are stored 565, followed by a check if all SMD data has been thoroughly 
processed i.e. the completion of the probing phase for possible Sequences 570. 

In addition there are a number of helpful characteristics that assist the analysis 
process. For example, the presence of Transport GuiObjs may be used such as Cancel and 

15 OK that are helpful in identifying profiles of any Action Structures series for Sequence 
identification. Another characteristic is the Host App Main Ref State, which acts as a known 
stasis point that is usually the start-up state of an application. Being the first "stable" 
processing point reached after first invoking the program and becoming ready for user 
modification of the primary file, the Main Ref State is a stable position from which to launch 

20 CHA Sequences into the depths of the program. Another set of identifiers are Monitor 
Record Markers embedded by user actions e.g. a keystroke, that mark particular GuiObjs as 
potential Target GuiObjs. Furthermore, queries can collect information from the user that can 
be used as clues to help identify targets. For example, the user may list keyword terms such 
as "Fonts", or "Colors" that can be searched in ChaDB and mapped to GuiObjs which in turn, 

25 can be detected in the Session Monitor Data (SMD). Once patterns have been marked as 
potential candidates for Sequences, the user is prompted for interactive selection of the types 
of output to be generated. After determining and sorting out their choices, Sequence 
construction processing can begin. 

Once the GuiObj or GiiiObj Group are identified during analysis of the SMD 

30 set, the Host App Model and Sequence generating code can be used. Referring to FIGS. 12- 
14 and 14A, the CHA Monitor's Auto Sequence Generation shares significant portions of the 

61 



WO 00/68769 PCT/US00/12106 

Dynamic Guide Sequence process in which the sequence entry point, upon completion of the 
Monitor Analysis phase, is found at 2123 in FIG. 13. Given the Potential Sequence Set 
defined by a series of Action Structures framed by Start and End points, the processing of 
Auto Sequence Generation begins by preparing the input data for Guide Sequence execution 
5 575. Using the GuiObj IDs and Keys embedded in the Action Structures, lookups 580 are 
performed in the Host App Model using functions from the Host App Model Processor 
Module 2040, shown in FIG. 12 such as cFindGuiObj and cFindPath. In some cases, the 
input Action Structures may include a single entry i.e. the Target GuiObj. With the Target 
GuiObj as the target, the Host App Model deduces the proper path from a given start point 

10 585 such as the Main Ref State position. This default start point position can also be user- 
controlled or modified via an editing tool interface. All this pre-processed and collected data 
becomes input preparation for the Dynamic processing flow that follows i.e. the module 
executes a Return 590 and resumes processing as described in FIG. 13 "Dynamic Fixed 
Guide Sequence Processing" at 2123. 

15 The problems in the prior art of working with copious amounts of confusing 

data are obviated by the disclosed CHA system and method by using a Monitor Tool that 
provides various processing and filtering functions to compile useful sequences. These 
sequences can be collected for later random access and compilation of a knowledge/action 
foundation for productive use in the Host Application. The Monitor Tool represents Action 

20 Events symbolically as icons with associated data windows that display a set of parameters 
associated with the Action event. As the user interacts with the tool's symbolic interface, 
messages can be sent to the Host Application and thus demonstrate on the live Host App the 
particular portions of the GUI interface which are controlled or associated with a selected 
Action Event. Furthermore, by interactively using filtering and other processes of 

25 elimination, viewing this representation of the Host App allows the user to either incorporate 
or eliminate members from the list of symbolic Action Events. Each Sequence identified 
from the previous analysis stage can be executed and demonstrated for the user's verification 
and subsequently exported to the ICAP list for later access. ICAP procedures located in 
ICAP Lists can be further accessed, combined or edited. An additional feature provided in 

30 the Monitor Tool is a Metrics feature that assesses the use of the Host App and retains 
statistics on usage for reporting on demand. 
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For usage analysis and automated procedure building, the CHA Monitor 
system can be applied to any GUI-based application program in any operating system 
including web browser applications for the Internet, intranets, extranets, etc.. The use of the 
CHA System and method to web browsing enables the analysis of any program or web page, 
5 and the generation of reports and statistics about how often and in what manner interface 
objects are used as well as methods for facilitating or accelerating interaction. This concludes 
the discussion on CHA Monitoring. 

A CHA Development Environment (CDE) Tool is provided which has a 
number of important functions. Based on the infrastructure as previously described, the CDE 
10 Tool is used to create instances of the various data types and to package and integrate them 
into the ChaDB, including the Host App Model, ready for delivery on any platform to support 
Dynamic Guide Sequences. Furthermore, the CDE Tool is also used for the creation of Fixed 
Guide Sequences. 

The CDE Tool main interface includes a Main window and a Production 

15 Toolbar window, in which the Main window includes all of the central interfaces for control 
of all processes and modes. Additionally there are also interfaces for setting global settings, 
various global object properties, and data file management facilities. The CDE Tool runs in 
both multiple networked mode on a network of computers as well as in single computer 
mode. The development network operation mode is most efficient to permit the running of 

20 the Host Application on a dedicated machine while the CDE Tool interfaces, debugging, and 
monitoring functions can run on their own dedicated computer resources. This has the added 
advantage of making all CHA Run Time processes transparent to the Host Application and of 
having minimal impact on the operation of the Host Application. 

The CDE Tool attaches InfoObjs to any of the Host App's GuiObjs and 

25 integrates the attached InfoObjs into the ChaDB. This tool allows for any number of different 
InfoObj resources to be attached to any GuiObj. While referencing the Host App Model, the 
InfoObj Attachment Process is performed by selecting the AttachlnfoObj Button from the 
CDE Tool's Main Window and having the cursor. Concurrently, status captions and displays 
reflect the AttachlnfoObj Mode of operation. A developer points to any Host App's GuiObj, 

30 and "locks" or selects the GuiObj with the Lock Control. The user may not be allowed to 
actuate other GuiObjs until the current selected GuiObj is unlocked. Concurrently, status 
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indicators display an icon that indicates the mode, and any known property information 
displayed in text about the locked GuiObj. 

Numerous selections appear in a GUI interface Resource List, and either 
through prompts or interface objects, code is invoked that injects into the record of the current 

5 Locked GuiObj one of the following selections: an InfoObj of different types Text and 
correspondent Voice Renderings, Static Illustration, Animation, or Interactive type. An 
Interactive Object Type may be implemented in many configurations, ranging from a simple 
implementation to an advanced level of complexity of interactivity to offer a range of 
interactions from simple to very complex involving extensive animation and rapid reaction to 

10 user input in the control of many tasks. This Interactive Object can appear in the Adjunct 
Window. The CHA system and method proceeds to determine if a resource item does not 
exist or is unavailable for use. If so, the user can create or enter a new instance of any of the 
previous InfoObj types by entering a text object, by recording a voice file, or by invoking a 
more complex tool Interactive Development Tool used for creating Interactive objects and 

15 activities. 

Upon creation, the user names or labels the resource and actuates the Submit 
InfoObj Button which then attaches the InfoObj to the current GuiObj record. The tool also 
enters one or more parameters as determined by the type, and the user also has the option to 
specify and/or input additional parameters or parameter values for the selected InfoObj. 

20 Additional Expression Behavior, i.e. properties rendered in the Expression Engine, are 
controlled by parameters for different InfoObj types such as Spatial Properties and 
Positioning information. Other expression-related parameters include Relative Offsets for an 
object's origin, Graphics Behaviors, Appearance Style, Wipe Style, Scrolling, Scheduling a 
time interval for appearance, and Interactive which includes waiting for a user to actuate or 

25 control an expression to disappear. Other property categories of the current Locked GuiObj 
include GuiObj Highlighting Style, and GuiObj Highlighting enabling. Using the interface 
provided in the CDE, the user then selects from the above options until the user determines 
that the current Locked targeted GuiObj is sufficiently complete or specified. When all 
parameters and/or parameter values are specified and/or input, the current GuiObj is 

30 Unlocked and the next GuiObj is processed until the Host App's support development is 
completed. 
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Additionally, the CDE Tool also includes a Fixed Sequence Creation 
Command that organizes and permits the execution of a number of procedural steps for 
building Fixed Guide Sequences. From the Guide Sequence Building interface controls, the 
Sequence New button is activated to begin building the sequence. When the user actuates the 

5 targeted GuiObj, a window appears which indicates a categorized list of available InfoObj 
resources that have been built into the selected GuiObj record during the InfoObj Attachment 
session. If there is only one InfoObj, then the InfoObj automatically appears as the user 
would actually view the InfoObj. Between multiple InfoObj s, there can be a variety of 
relationships. For example, more than one InfoObj resource may be joined together or 

10 associated with each other using the Chain command. An InfoObj can be invoked 

conditionally, depending on an internal interpreter, a user input or setting e.g. InfoObj Detail 
Level, or a Host App's action. More than one InfoObj can be displayed concurrently e.g. 
Text and Graphic InfoObjs. Each InfoObj can be set to display, for a specific time interval, 
InfoObjs having animation properties such as moving across the screen or drawing in 

15 different wipe styles e.g. the object fades in. 

Generally the CHA System provides a run-time platform for the capture and 
recreation of an application's states that can be delivered remotely through known network 
techniques. Located on the CHA Server, the ChaDB Server Component using one or more 
CHA Databases, determines targets and pathways to be compiled into a CHA System output 

20 i.e. as Fixed or Dynamic Guide Sequences or as ICAP solutions. The Guide Sequences are 
converted into at least one of a set of selectable compact Assistance Execution Formats 
configured for different solutions which can be transmitted over networks of various 
bandwidths to client computers hosting CHA System run-time components. 

There are numerous applications for a CHA Server that provides various 

25 Assistance Execution Formats. For example, in Collaborative Activities, users trade CHA 
Sequences or post such CHA Sequences for others users to use. Also, Training may be 
implemented with the CHA Server, allowing an on-line live tutor to monitor user progress, or 
for allowing a user to retrieve Tech Support for individual application or system problems. 
Other applications include user learning remote execution of Guide Sequences for example, 

30 for live tech support; collaborative activities and applications; live interaction with other 
users, and instruction. 
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The CHA server offers an extensive CHA Database of assistance that can be 
selectively downloaded and/or accessed on demand. Such assistance CHA sequences can be 
accessed and the user may be charged with a fee by extent or duration of use, and customers 
may be permitted to rent CHA sequences for a prescribed period of time. The CHA Server 

5 implements a number of functions which include the determination of the profile of the user 
whose access privileges are dependent on criteria e.g. a payment plan; the notification and the 
updating of client resources and components; and the tracking of which CHA run-time 
components and resources reside on a user's system. 

The CHA System Guide Sequences and Procedures alleviate the problem of 

10 real-time synchronization of events and the display of such events between two computers 
over narrow networked band widths in known remote control software systems. The CHA 
System alternative, implementing the capture, modification, and delivery of Sequences, 
makes an effective support mechanism for users. For example, a technical support person can 
send the user Guide Sequences to execute a software fix or to lead the user through a 

15 procedure to rectify a technical problem in the user's computer. In another scenario, a user 1 
can capture a set of actions and convert such actions into a Sequence that can be sent to a 
support person for analysis. The tech support "plays" or simulates the Sequence, makes a 
correction or otherwise modifies the Sequence, and then returns the corrected Sequence to the 
user who then executes the Sequences as the solution to a particular problem or query. 

20 Auxiliary data or instructions can be easily embedded into the solution Sequence. 
Alternatively a user can also browse or search through lists of solution Sequences in a 
compact form, download a selection of such solution Sequences, and execute any specific 
Sequence as needed or desired, or else at least one solution Sequence can be requested by the 
user, and E-mailed or otherwise delivered to the user for implementation. Furthermore, the 

25 tech support person may be able to activate a request for an Assistance Topic which is then 
generated and forwarded to a user via E-mail or other software delivery methods. 

Overall, the CHA system and method may be used to execute a finely tuned, 
interleaving on-line delivery of assistance information, which may include software as a 
solution Sequence, to the user and to provide the delivery mechanism needed to make CHA 

30 work under stringent or demanding network conditions. The CHA system and method 
achieve such results because Guide Sequences are partitioned into segments called SeqSteps, 
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in which each SeqStep can be delivered as a separate, asynchronous data stream. By queuing 
up future SeqSteps, subsequent SeqSteps can be delivered asynchronously while executing 
current ones to provide a more seamless, wait-less user experience. 

FIG. 16 illustrates the server processing of a request for an Assistance Topic 

5 made from a client computer. As shown in step 2605, the user interacts with any of a number 
of search tools, and generates a request Rn for an Assistance Topic in step 2610. Once 
generated, using infrastructure mechanisms for remote transmission e.g. WinSocket or 
Hypertext Transfer Protocol (HTTP) in a browser, etc., Rn is transmitted to the CHA Host 
Application (HA) Server in step 261 5. On the HA Server Rn is received and processed into 

10 form RnQ in step 2620, which unpacks the request and prepares the received input and its 
parameter set for processing. The RnQ is passed on to the Main Host App Server 
Components in step 2625, and parsing and processing begins in step 2630 with a test to 
determine if the request Rn has been previously processed via indexed searching and whether 
the request Rn is currently in the HA Server's storage. If the request Rn is currently stored, 

15 then Rn is transformed into an output Sn for transmission in step 2655, and a retrieval takes 
place for the pre-packed delivery components which include all needed code and data for the 
Guide Sequence. If the request Rn is not currently stored, then step 2635 is performed in 
which RnQ is evaluated and passed to the Host App Model Processor 2040, shown in FIG. 
12, in which RnQ is then entered into the different Host App Model function set for execution 

20 Guide Sequence generation in step 2640. The output Sn is then formed in step 2645 either 
from a dynamically generated Guide Sequence which has been created, and then assembled, 
or retrieved from the Application CHA Database. 

The output Sn includes Host App Model functions and supporting InfoObjs 
and Commentary Objects as well as opcode operators to facilitate the synchronization of user 

25 actions and the processing of objects in the expression engine. In step 2650, the output Sn is 
compressed, packed, and prepared for transmission. The output Sn is then passed to the 
server transmitting module in step 2655, and the output Sn is conveyed through the network 
connection to arrive intact on the client machine in step 2660. Upon receipt, the output Sn is 
expanded and unpacked at in step 2665, and passed to the client computer's local CHA Run- 

30 Time components in step 2670, where the Guide Sequence is executed in step 2675, as 
described herein for both fixed and dynamic Guide Sequence processing. 
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Using the CHA system and method in conjunction with live host applications, 
the CHA System and method has the ability to quickly generate dynamic paths and to 
synchronize associated InfoObjs accurately, as well as having the ability to generate 
Commentary and Connective language elements between and leading to user actions. Every 
5 path generated and executed can have a unique combination of elements. 
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WHAT IS CLAIMED IS: 

1 . A computer-based assistance system for providing operational guidance 
of commands to use a computer program, the assistance system comprising: 

a command indicator for visually indicating to a user a portion of a 
5 display of the computer program corresponding to a specific command to be executed; and 
an interactive assistance object, responsive to the command indicator 
indicating the specific command, for interacting with the user to guide the user in execution 
of the specific command. 

2. The assistance system of claim 1 , wherein the interactive assistance object 
10 includes an animation generator for generating an animated character to visually interact with 

and guide the user to execute the indicated specific command. 

3 . The assistance system of claim 2, wherein the animation generator 
generates a plurality of animated characters to visually interact with each other to guide the 
user to execute the indicated specific command. 

15 4. The assistance system of claim 1 , wherein the interactive assistance object 

includes a text message generator for displaying on the display a text message associated with 
the indicated specific command to guide the user to execute the indicated specific command. 

5. The assistance system of claim 1, wherein the interactive assistance object 
includes an audio message generator for generating audio prompts to audibly interact with 

20 and guide the user to execute the indicated specific command. 

6. The assistance system of claim 1 , wherein the interactive assistance object 
adaptively responds to user inputs to continually guide the user to execute the indicated 
specific command. 

7. The assistance system of claim 6, wherein the computer program operates 
25 with the user through a graphic user interface (GUI), including movements and actuations of 

a current screen position indicator (CSPI); and 

wherein the interactive assistance object adaptively responds to 
movement of the CSPI on the GUI to guide the user to execute the indicated specific 
command. 

30 8. The assistance system of claim 7, wherein the CSPI is a cursor. 
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9. The assistance system of claim 8, wherein the CSPI moves in response to 
corresponding movements of a mouse device. 

10. The assistance system of claim 9, wherein the interactive assistance object 
responds to the cursor movement by prompting the user to move the mouse and thereby the 

5 cursor to the indicated specific command displayed on the GUI. 

1 1 . The assistance system of claim 1 , wherein the command indicator visually 
indicates the portion of the display by focusing the attention of the user to a predetermined 
region of the display surrounding the specific command. 

12. The assistance system of claim 1 1, wherein the focusing includes 

10 overlaying a rectangular box as the predetermined region on the display of the computer 
program with the box surrounding the specific command of the computer program. 

13. The assistance system of claim 11, wherein the computer program 
represents commands by corresponding actuatable regions on the display; and 

wherein the focusing includes providing a substantially distinct 
15 appearance of the indicated portion of the display, with the distinct appearance being different 
from the appearance of the actuatable region associated with the displayed specific command. 

14. The assistance system of claim 13, wherein the appearance of the 
indicated portion includes a displayed color. 

15. The assistance system of claim 14, wherein the substantially distinct 

20 appearance of the indicating portion includes changing the displayed color to appear to flash. 

1 6. The assistance system of claim 13, wherein the appearance of the 
indicating portion includes a displayed shape. 

17. The assistance system of claim 16, wherein the substantially distinct 
appearance of the indicating portion includes providing an animated displayed shape for the 

25 indicating portion. 

1 8. A computer-based assistance system for providing operational guidance 
of commands to use a computer program, the assistance system comprising: 

a search tool for searching through a plurality of records representing a 
host application to determine at least one assistance item key mapping a sequence 
30 corresponding to respective controls for implementing a particular command; and 
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a sequence processor, responsive to the at least one assistance item key, 
for implementing the particular command. 

19. The computer-based assistance system of claim 18, wherein the sequence 
processor processes the plurality of assistance item keys in synchronization with user-driven 

5 events. 

20. The computer-based assistance system of claim 18, wherein the at least 
one assistance item keys is fixed in a predetermined order. 

21 . The computer-based assistance system of claim 1 8, wherein the at least 
one assistance item key is dynamically generated in response to the user-driven events. 

10 22. The computer-based assistance system of claim 1 8, wherein the user 

selects the first search tool from a plurality of available search tools. 

23. The computer-based assistance system of claim 18, further comprising: 

a commentary generator, responsive to the processing of each control, for 
generating an available commentary to the user corresponding to the processing of the 
15 respective control. 

24. The computer-based assistance system of claim 23, wherein the 
commentary generator generates visual messages as the commentary for output to the user 
through a display. 

25. The computer-based assistance system of claim 24, wherein the 
20 commentary generator generates animation as the visual messages. 

26. The computer-based assistance system of claim 23, wherein the 
commentary generator generates audio messages as the commentary for output to the user 
through a speaker. 

27. A method for providing dynamic operational guidance of commands to 
25 use a computer program, the method comprising the steps of: 

iteratively searching a Host Application Model to locate a target graphic 
user interface object (GuiObj) corresponding to a command to execute, and to locate a current 
location of a user in the Host Application Model; 

determining a path through the Host Application Model from the target 
30 GuiObj to the current location of the user; and 



71 



WO 00/68769 



PCT/US00/12106 



generating a dynamic guide sequence record from the path for executing 

the command. 

28. The method of claim 27, further comprising the steps of: 
receiving user inputs corresponding to a selection of the target GuiObj 

5 associated with the command; 

generating a GuiObj identifier (ID) corresponding to the selected target 

GuiObj; and 

mapping the GuiObj ID to a GuiObj key; 

wherein the step of iteratively searching a Host Application Model 
10 includes the step of: 

searching the Host Application Model using the GuiObj key. 

29. The method of claim 27, wherein the step of determining the path 
includes the step of: 

apply path finding techniques to the Host Application Model to find a 
15 best path through the Host Application Model. 

30. The method of claim 29, wherein the step of applying path finding 
techniques is performed in real-time to dynamically determine the best path and to execute 
the command in real-time. 

3 1 . The method of claim 27, fiirther comprising the step of: 

20 retrieving information objects (InfoObjs) based on a portion of the path; 

and 

outputting the InfoObjs to the user. 

32. The method of claim 3 1 , wherein the step of retrieving InfoObjs includes 

the step of: 

25 retrieving a predetermined detail level of the InfoObjs for output to the 

user. 

33 . The method of claim 3 1 , wherein the step of outputting the InfoObjs 
includes the step of: 

generating visual messages from the InfoObjs for output to the user 

30 through a display. 
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34. The method of claim 33, wherein the step of generating the visual 
messages includes the step of: 

generating animation as the visual messages. 

35. Themethodof claim 31, wherein the step of outputting the InfoObjs 

5 includes the step of: 

generating audio messages as the InfoObjs for output to the user through a 

speaker. 
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